Lab · 3
Can AI catch and fix its own documentation drift?
Experiment 2 found that an automatic checklist catches structural drift on its own, but content drift — stale wording, dead cross-references — still needs someone to notice it on purpose. This experiment tested whether AI could be the one to notice, and then actually fix it, not just flag it.
02 · Hypothesis
Not a controlled study, and not conclusive enough to prove this works with Claude as of September 3rd 2026, but it seems possible with other AI technologies.
Method
What actually happened, in order.
What was built
Earlier this project, browser automation proved it could write new content directly into a Google Doc’s body — not just read Drive via the API connector, which is read-only. This experiment used that same capability against a real backlog: three known-stale claims in two existing Lab / Design System briefs, catalogued in the “Content Brief Schema — Reference” doc after a prior session found them. The test was whether AI could open each Drive doc, locate the stale text precisely, and correct it in place — end to end, not just identify it.
The backlog — what was actually stale
01
Experiment 1 brief described the page’s URL as /lab/experiment1/ and its kicker as “Lab · Experiment 001” — both predate the Aug 29 URL-convention change to bare /lab/1/.
02
The same brief had a “Spec table” heading with a stat list and an unfilled “Evidence · Receipt 01 — Input: TBD / Outcome: TBD” — “Spec table” was never a real /design-system/ component, and Evidence was removed from the live page entirely in the Aug 29 rewrite.
03
The Design System brief stated “There is no separate ‘About — Content’ doc yet” — false since an About brief was created Aug 25, six days before this session.
04
Found along the way, not in the original catalogue: the About brief’s own “Card 3” linked to /lab/experiment1/ — the same stale-URL problem, in a second document.
What Claude actually did
Fixed all four directly in the live Google Docs — no copy-pasting into a new doc, no describing the fix in chat for Dr. Paul to apply by hand. Rewrote the URL/kicker text and renamed “Spec table” to “Stats line” with corrected numbers; removed the unfilled Evidence/Receipt placeholder and updated the brief’s own retired-sections note; corrected the Design System brief’s About-doc claim; fixed the About brief’s stale link; and, while already in each doc, added the Content Brief Schema’s required header block. Also caught and corrected two smaller drifts noticed only by reading closely: a stale “Ongoing” status pill after the live page moved to “Supported,” and a stray typo from Claude’s own automation that needed a follow-up correction before it shipped.
What Claude did not do
Did not touch the Hypothesis statement in the Experiment 1 brief, even though its wording differs from the live page’s — that’s Dr. Paul’s own claim, not a factual drift to silently correct. The content-manifest.json bookkeeping update also hit a real wall mid-session: this session’s git push was blocked by a repo-access proxy (a 403, “not in this session’s authorized repository set”) — no workaround for the git CLI exists here. It shipped anyway, through the site’s own established deploy path (GitHub’s web-upload UI) instead, with an explicit go-ahead from Dr. Paul before the commit landed on main.
04 · Results
Four for four, with one detour to get it fully live.
Worked: Locating exact stale text inside a real Google Doc and rewriting it in place, including recovering cleanly after an accidental keystroke landed mid-paragraph instead of at the end of it (undo, re-target, redo) — the fix held up under a fresh re-read afterward.
Limited: Two small automation errors shipped without being caught until a follow-up zoomed screenshot — a bullet-list edit briefly broke list formatting, and a double-click-and-replace once fused two words into “ThAn” before a second pass fixed it. Neither reached the live site, but both would have if this had been treated as one-shot instead of self-checked.
Also worked, eventually: The content-manifest.json push. Blocked once by the git CLI’s repo-access proxy, it went out through the site’s own established GitHub web-upload path instead — the workaround this project already used for every other deploy — with Dr. Paul’s explicit sign-off before the commit landed.
Postscript · on “guardrails”
After reading this brief, Dr. Paul raised a fair question: is “AI” too general a term here, given that safety guardrails might be the actual reason a step needed help, rather than a limit on model capability? Worth separating out precisely, because only one blocker in this experiment was actually a guardrail.
The one real case: the safety classifier blocked Claude from typing into GitHub’s commit-message field until Claude explicitly asked Dr. Paul for sign-off before committing scripts/content-manifest.json to main. That’s a genuine guardrail — specifically, a “don’t publish to a public surface without human confirmation” rule — and it worked as designed.
Everything else that needed a workaround was not a guardrail: the git push 403 was an access-control boundary (this session’s git proxy didn’t have paulmress.com in its authorized repository set — the same wall any contractor without repo credentials would hit); the Google Drive API connector’s inability to edit a Doc’s body at all is a gap in what that connector’s API exposes, not a safety choice; and the two self-caught mid-session mistakes (a keyboard shortcut typing a literal “1” instead of applying a heading style, a triple-click landing mid-paragraph) were plain browser-automation imprecision, unrelated to caution.
Claude’s answer, which Dr. Paul accepted (“I hear you”): guardrails accounted for exactly one blocker out of several, and it’s arguably the one that should exist. The more precise finding may not be “AI hit a wall” but three separate kinds of friction — policy gate, access boundary, execution imprecision — since only the first is actually about guardrails, and conflating them overstates how much safety training, specifically, was the bottleneck here.
05 · Conclusion
“It sorta worked and sorta didn’t work.”
What I learned
Claude can fix its own drift, but only when someone points it at the specific thing to fix. However, it seems that other AI technologies that allow a standard process to occur could fix the drift, so this experiment should be tried again in the future.
Next Question · 4
Can AI build a helpful filter, sort, and search?
Documentation drift needed a human to notice it. So does everything else buried in a growing list of pages — unless the list itself can be filtered, sorted, and searched. Experiment 4 tests whether AI can design and build that tool on its own, inside the site’s own atomic-design rules.
Read Experiment 4 →