Lab · 3
Can AI catch and fix its own documentation drift?
Experiment 2 found that an automatic checklist catches structural drift on its own, but content drift — stale wording, dead cross-references — still needs someone to notice it on purpose. This experiment tested whether AI could be the one to notice, and then actually fix it, not just flag it.
Hypothesis
AI can find and directly fix its own stale documentation in Drive, the same way it already fixes stale code — without a human doing the editing by hand.
Finding
It sorta worked and sorta didn’t work — Claude can fix its own drift, but only when someone points it at the specific thing to fix.
Not a controlled study, and not conclusive enough to prove this works with Claude as of September 3rd 2026, but it seems possible with other AI technologies.
Method
Catalogue
Three known-stale claims, found earlier and logged in the Content Brief Schema doc.
Locate
Claude opened each Google Doc and found the exact stale text.
Fix
Edited the text directly in Drive — no copy-pasting into a new doc for Dr. Paul to apply by hand.
Verify
Re-read each doc afterward to confirm the fix actually held.
Tech
The experiment
Four for four, with one detour to get it fully live.
AI handled
- Locating and rewriting four stale items directly in three live Google Docs
- Renaming a “Spec table” heading to “Stats line,” with corrected numbers
- Removing an unfilled Evidence placeholder and updating the retired-sections note
- Adding the Content Brief Schema’s required header block to each doc touched
Dr. Paul handled
- Deciding which drift was real vs. his own claim to leave alone (Experiment 1’s hypothesis wording)
- Signing off before the
content-manifest.jsoncommit landed on main - Raising the guardrails question below, after reading the draft
- Final review
Worked: Locating exact stale text inside a real Google Doc and rewriting it in place, including recovering cleanly after an accidental keystroke landed mid-paragraph instead of at the end of it — the fix held up under a fresh re-read afterward.
Constraint: Claude can fix drift once it’s pointed at the exact thing to fix, but doesn’t yet notice new drift on its own — two small automation errors also shipped uncaught until a follow-up screenshot, though neither reached the live site.
Postscript · on “guardrails”
After reading this brief, Dr. Paul raised a fair question: is “AI” too general a term here, given that safety guardrails might be the actual reason a step needed help, rather than a limit on model capability? Worth separating out precisely, because only one blocker in this experiment was actually a guardrail.
The one real case: the safety classifier blocked Claude from typing into GitHub’s commit-message field until Claude explicitly asked Dr. Paul for sign-off before committing scripts/content-manifest.json to main. That’s a genuine guardrail — specifically, a “don’t publish to a public surface without human confirmation” rule — and it worked as designed.
Everything else that needed a workaround was not a guardrail: the git push 403 was an access-control boundary (this session’s git proxy didn’t have paulmress.com in its authorized repository set — the same wall any contractor without repo credentials would hit); the Google Drive API connector’s inability to edit a Doc’s body at all is a gap in what that connector’s API exposes, not a safety choice; and the two self-caught mid-session mistakes (a keyboard shortcut typing a literal “1” instead of applying a heading style, a triple-click landing mid-paragraph) were plain browser-automation imprecision, unrelated to caution.
Claude’s answer, which Dr. Paul accepted (“I hear you”): guardrails accounted for exactly one blocker out of several, and it’s arguably the one that should exist. The more precise finding may not be “AI hit a wall” but three separate kinds of friction — policy gate, access boundary, execution imprecision — since only the first is actually about guardrails, and conflating them overstates how much safety training, specifically, was the bottleneck here.
Finding
It sorta worked and sorta didn’t work.
Claude can fix its own drift, but only when someone points it at the specific thing to fix.
Other AI technologies that allow a standard process to run might close that gap — worth trying again as the tooling matures, rather than treating this one attempt as the final word.
Implication
For now, content-drift review isn’t something to hand fully to AI and walk away from. It’s a real productivity gain once a human has already spotted the problem — but the noticing still has to be a person’s job.
Point AI at exactly what’s broken, and it’ll fix it. It won’t yet go looking on its own.
Next Question · 4
Can AI build a helpful filter, sort, and search?
Documentation drift needed a human to notice it. So does everything else buried in a growing list of pages — unless the list itself can be filtered, sorted, and searched. Experiment 4 tests whether AI can design and build that tool on its own, inside the site’s own atomic-design rules.
Read Experiment 4 →