Lab · 8

Can AI create a pre-deploy checker that replaces manual accessibility review?

Every experiment so far has tested what AI can build for the site. This one turned that same question inward — could AI build a tool that audits the site's own accessibility before anything ships, catch real problems, and prove it by actually fixing what it found?

Supported Updated Sept. 9, 2026

Hypothesis

Can AI create a pre-deploy checker that replaces manual accessibility review?

Finding

The checker found bugs that were fixed. It seems like a great use case for AI.

Tested by building two accessibility-checking scripts and running them against the real, live production site both before and after fixing what they found — not a synthetic benchmark, and not a study across other sites or design systems.

Method

Traditional

Ship pages as they come togetherNo structured accessibility checkIssues surface only if someone complains

Experiment

Build a pre-deploy checkerRun it against the live siteFix every verified issueRe-run to confirm

Build

Wrote two Python scripts covering WCAG contrast, touch-target size, heading hierarchy, and unlabeled form controls — a static HTML parser and a Playwright-rendered checker.

Audit

Ran both scripts against the real, live site and separated genuine bugs from checker blind spots.

Fix

Shipped verified fixes for four real findings across two live pages, then deployed.

Verify

Re-ran both checker scripts against the live site after deploying and confirmed zero remaining findings.

Tech

Claude Python Playwright

The experiment

Four real, verified problems, four fixes shipped, and one technique the checker itself caught as not actually working.

AI handled

Building both checker scripts from WCAG success criteria: contrast (1.4.3), touch-target size (2.5.8), heading hierarchy, and label association
Running them against the live site and separating real bugs from a checker blind spot — WCAG 2.5.8’s inline-text exception, which the checker doesn’t know about and always flags
Trying an invisible ::before hit-area expander for the touch-target fixes first, discovering via the checker’s own re-run that it didn’t change what the browser actually measures, and abandoning it for real padding plus a matching negative margin instead
Fixing and deploying all four real findings, then re-running both scripts to confirm zero remaining issues

Dr. Paul handled

Requesting the checker be built first as an internal, non-user-facing pre-deploy gate — not a public page
Giving explicit go-ahead to fix and push the real findings live
Confirming the hypothesis, the finding, and the status pill in his own words

4 real bugs found · 4 fixes shipped · 3 deploy commits · 2 checker scripts (static + Playwright-rendered) · 1 technique tried and abandoned once the checker proved it didn’t work

Worked: The checker caught real, previously-unknown bugs already live on the site — two unlabeled sort selects, a heading-hierarchy skip, three low-contrast decorative dividers/arrows, one low-contrast status pill, and four undersized touch targets — and verified every fix by re-measurement, not by eye.

Constraint: The first attempt at the touch-target fix looked correct in the browser, but the checker’s own re-run showed the measured element box hadn’t changed — real padding and a negative margin were needed instead. The checker also doesn’t know WCAG 2.5.8’s inline-text exception, so a few flagged links still needed a human to read the surrounding HTML before ruling them out as non-bugs.

Finding

The checker found bugs that were fixed. It seems like a great use case for AI.

Two rendering-based scripts caught four real, previously-shipped accessibility bugs across four live pages that had gone unnoticed until this experiment.

Every fix was verified by re-running the same scripts against the live site after deploying, not just checked by eye.

Next Question · 9

What’s next is still open.

Not decided yet — check back.

Back to the Lab