Lab · 2
If AI replaced the CMS, what stops the site from drifting?
Experiment 1 found that AI can replace a content management system (CMS). But a CMS quietly does something else too: it keeps a site consistent, page after page, without anyone having to think about it. Once AI took over and the CMS was gone, Dr. Paul tested whether a written design rulebook, plus a simple automatic checklist, could do that job instead.
Hypothesis
A design rulebook, backed by a simple automatic check, can keep an AI-built website consistent — doing the job a CMS normally does, without a CMS in place.
Finding
Some of the job, yes. Small style details — colors, spacing, fonts — update everywhere at once, because every page pulls them from one shared file, so that part fixes itself. But the CMS's other job, catching bigger mistakes before they ship, doesn't happen on its own. Someone, or something, still has to notice those on purpose.
Some mistakes were caught on this site — not a controlled study, and not enough to prove this works everywhere. But it worked to an extent.
Method
Launch
A new page or change — often drafted by AI — goes live. Nothing else automatically stops a mistake from shipping.
Drift
Something on the site quietly stops matching the rulebook — the wrong color, a leftover sentence, an inconsistent footer.
Check
Before anything goes live, an automatic checklist scans every page for exactly this kind of mistake — the job the CMS used to do.
Fix
A human or AI reads the results and fixes the problem by hand — the checklist finds mistakes, it doesn't fix them.
Tech
The experiment
The artifact isn’t a writeup — it’s the automatic checklist itself (scripts/audit-site.py), watching over the same AI-built site from Experiment 1, plus the two real mistakes it’s already helped catch, described in the design system’s own rules.
AI handled
- Writing the checklist
- Spotting the mistakes
- Drafting the fixes
Dr. Paul handled
- Deciding what counts as a mistake
- Approving the fixes
- Final review
Worked: The automatic checklist reliably catches the bigger mistakes — it's flagged nothing since, and stayed clean even while this very page was being built.
Constraint: It can't catch mistakes in the writing itself — repeated wording, outdated phrasing, a wrong reference. That needed a separate tool built just for text.
Finding
Replacing the CMS didn’t remove the job.
It just left it unassigned.
Small style details fix themselves automatically — colors, spacing, and fonts update everywhere at once, because every page pulls them from the same shared style file.
Bigger mistakes aren’t automatic — a leftover sentence sat in one page’s footer until a person noticed it. That’s exactly why an automatic checklist now runs before every page goes live, instead of relying on memory.
Implication
Swapping AI in for a CMS doesn’t just remove a tool — it removes whatever that tool was quietly enforcing. Something has to pick that job back up, on purpose.
If you replace the CMS, replace what it was enforcing too. A design rulebook only works if something keeps checking it against what actually went live.
Next Question · 3
Should Dr. Paul become an avatar?
Not this experiment's question, but the next one in the queue, raised back in Experiment 1: talking avatar, text chatbot, or shelved entirely.
Back to the Lab