Lab · 10
Can AI catch its own mistakes automatically, instead of waiting to be told?
Experiment 3 found that AI could fix a mistake on the website — but only once someone found it first and pointed it out. This experiment asks the same question again, with one change: instead of AI fixing what it’s told to fix, we built something that checks the website on its own, every time it changes, without anyone having to ask.
Hypothesis
Can a check that runs on its own — automatically, every time the site changes — succeed where Experiment 3’s one-time fix couldn’t?
Finding
A process that checks itself found a mistake Experiment 3’s one-time fix never could.
This has only run for real once, on one page. It’s not a controlled study, and it isn’t proof that every check on the site should work this way — other candidates for the same treatment (checking the rest of the site’s pages, not just this one) haven’t been tried yet.
Method
Checklist
Instead of waiting for someone to notice a problem, we listed every claim one page makes about the rest of the site — which pages use which features, how many experiments exist, and so on.
Automatic
Turned that checklist into a standing process that runs every time the site changes, instead of something someone has to remember to ask for.
Test
The first real run happened right after two new experiments went live, before anyone had manually checked whether the page’s claims still matched.
Double-check
Anything flagged got checked a second time before it was trusted. Two real mistakes were confirmed and fixed. One case wasn’t clear-cut, so it was handed to Dr. Paul instead of guessed at.
Tech
The experiment
Same question as Experiment 3 — tested with a different approach this time.
AI handled
- Building the automatic check and turning it into a standing process that runs on its own
- Running it for real, right after two new experiments shipped without a follow-up check
- Finding two real mistakes and one unclear case
- Fixing the two real mistakes and flagging the unclear one instead of guessing at it
Dr. Paul handled
- Picking this idea, from a larger plan, to actually test
- Deciding what to do with the one unclear case
- Approving the fix before it went live
- Asking for this check to keep running automatically from now on, and for this write-up to be simplified until it was easy to follow
Worked: The process found two real mistakes nobody had noticed, the very first time it ran for real — without anyone pointing AI at a specific problem first.
Constraint: This only checks one page so far. The rest of the site isn’t covered yet. And it still needed a person to make the one genuinely unclear call — the process finds and sorts problems, but it doesn’t decide the hard ones on its own.
Finding
A process that checks itself found a mistake Experiment 3’s one-time fix never could.
Experiment 3 could only fix what a person had already found and pointed at; this process found a mistake nobody had noticed.
It also knew its own limits — it double-checked anything it flagged, and asked a person instead of guessing on the one case that wasn’t clear-cut.
Implication
Experiment 3 ended by wondering if a smarter AI model might eventually solve this on its own. It turned out that wasn’t the answer. Nothing about the AI itself changed between these two experiments. What changed was building a better system around it — one that checks automatically instead of waiting to be asked, and still checks with a person before making an uncertain call. The lesson here isn’t that AI got smarter. It’s that a better process, built once, can keep working long after the person who built it stops thinking about it.
A better system, not a smarter AI, is what actually worked.
Next Question · 11
What’s next is still open.
A few ideas already on the table: checking the rest of the site’s pages the same automatic way, not just this one.
Back to the Lab