Test accessibility in an AI-built interface beyond a green axe scan

Combine automated accessibility checks with keyboard journeys and manual assistive-technology evaluation. Playwright can drive an interface into the state you want to examine, and axe can report the rules it detects there. A clean scan does not establish that someone can complete the task. W3C explicitly says automated tools cannot determine accessibility on their own. ^[1]

A hypothetical agent-built “Invite teammate” dialog makes the distinction concrete. Its button has a label, the text has sufficient contrast and the initial page scan is clean. But opening the dialog leaves keyboard focus behind the overlay. An invalid email produces a visible red message that the person using a screen reader never encounters. The screenshot looks finished while the invitation remains difficult to send.

The workflow below is NO FEED’s proposed acceptance process for that interface. It is a technical testing guide, not a legal compliance assessment or a claim that this hypothetical component has been tested.

Choose a task and its states before choosing a scanner

List what a person must do: find the invitation control, open it, enter an address, understand an error, correct it, send the invitation and confirm the outcome. Include cancellation. Each step creates a different state worth checking.

Playwright’s accessibility guide demonstrates running @axe-core/playwright against the rendered page and opening an interactive menu before scanning it. The scan observes the current state at invocation. Scanning the unopened dialog’s page cannot inspect content that has not been rendered yet. ^[2]

For the example, run an automated scan at initial load, with the dialog open and after the validation error appears. Record the tested route, application revision, fixture, browser and rule configuration. If you scan only the dialog region, state that boundary in the result; the rest of the page has not acquired coverage by association.

Ask the coding agent to return specific failures with affected controls and reproducing actions. “Make it accessible” gives the reviewer no reliable way to distinguish an implementation change from a successful user journey.

Use each check for the question it can answer

These are complementary evidence categories, not percentages of accessibility. W3C recommends combining user evaluation with standards-based assessment; user testing can reveal usability problems that conformance checking alone misses. ^[3]

A user report shows why the setup belongs in the result

In July, Reddit user StrollingGamer described JAWS navigation repeatedly jumping to the bottom of conversations in AI chat interfaces on Windows 11. Their follow-up said the problem appeared after only five or six exchanges and Claude read marginally better with NVDA. Replies described different experiences and suggested screen-reader or browser behavior as possible explanations. ^[4]

This is an attributed interaction problem, not evidence that those interfaces were AI-generated or that the same defect remains today. It demonstrates the useful level of a test report: task, assistive setup, observed failure and unresolved cause. A single favorable result from another setup cannot erase that account.

Follow focus through opening, errors and closing

The WAI-ARIA Authoring Practices modal-dialog pattern describes focus moving inside when a dialog opens, Tab and Shift+Tab staying within it, Escape closing it and focus normally returning to the invoking element. The pattern includes exceptions where another destination better fits the resulting workflow. It also discusses choosing initial focus based on the dialog’s content and consequences. ^[5]

Turn those expectations into an intentional keyboard journey for the example. Reach “Invite teammate” with the keyboard and activate it. Check where focus lands and whether its indicator is visible. Move forward and backward through the email field, submission control and cancellation control. Close the dialog and identify the next useful destination.

The generated test should explain why each expected focus target is appropriate. Hard-coding whichever element currently receives focus would preserve a bug as the baseline. Keep the behavioral expectation separate from the current implementation, especially when the same agent is editing both.

Also try the error path. After submission with an invalid address, can the person find the explanation and return to the field? The answer depends on the entire sequence, not the presence of one ARIA attribute. A screenshot or DOM dump can help diagnose the failure; neither substitutes for completing the journey.

Check the meaning of feedback, not only its existence

W3C’s form-notification guidance recommends clear error descriptions, identification of the affected control and instructions for correction. It also covers notifying users about dynamically inserted errors and confirming successful completion. ^[6]

For this invitation, “Invalid” is weaker than a message that tells the reader an email address is needed and associates the explanation with the field. The exact copy should reflect the product’s validation rule. Do not ask the agent to add a format requirement that the service does not actually impose.

In a manual screen-reader evaluation, record the browser, operating system, assistive technology and versions used. Listen to the control’s name, whether the dialog’s purpose is clear, what happens when the error appears and how success is communicated. A test asserting that an alert node exists checks markup; it does not record what a person heard in that setup.

When uncertainty remains, keep it attached to the task: “Invitation success feedback not yet evaluated with the supported screen reader” is actionable. “Accessibility passed” obscures the missing observation.

Keep exceptions small enough to review

An existing application may have unrelated violations. Playwright’s guide offers ways to constrain scans or manage known violations, but any exclusion changes the scope of the result. ^[2]

Our proposed rule is to give each accepted exception a named defect, affected state, owner and review date. If an agent disables a rule or excludes the dialog to obtain a pass, the patch must explain which coverage was removed. A lower violation count can result from testing less of the interface.

Review fixes with the same task and configuration used to expose the failure. Add another state when the change affects it; do not repeat broad scans simply to accumulate green output. Preserve useful failed evidence alongside the corrected result so someone can understand what changed.

For your next AI-generated component, choose one complete keyboard task with an error and a recovery step. Give the agent that acceptance journey before it starts coding, run automated checks in the relevant states, and reserve a manual assistive-technology check for the interaction that automation cannot settle.