Diagnose flaky tests before a coding agent “fixes” CI

The first attempt fails. The retry passes. Playwright classifies that result as flaky; the second outcome does not explain the first. ^[1]

Before a coding agent changes CI, preserve the failed attempt, reproduce under controlled conditions and identify the cause while retaining the assertion’s meaning. The task is to explain the inconsistent behavior, not simply produce another passing run.

In a hypothetical checkout suite, two workers reuse the same test account. One changes its address while another submits an order. The order test sometimes sees the wrong delivery address. Asked to “make CI green,” an agent might add a sleep, increase retries or remove the address check. Each can change the pipeline’s appearance while leaving the shared-state problem intact.

The task should instead be to explain the first failure and preserve the behavior the test protects. The investigation below is a proposed workflow, not a measured result from that hypothetical suite.

Keep the first failure before rerunning anything

Save the commit, test name, browser project, worker count, runner environment, fixture identity, error and retry number. Keep relevant traces and application logs with access appropriate to their contents. A trace can contain network request and response bodies; it belongs with test evidence, not an indiscriminate public issue attachment. ^[2]

Playwright’s common on-first-retry tracing setting records the retry, not the original attempt. If the retry passes, that trace may show a different sequence from the failure you are investigating. For a focused reproduction, record the original attempt using a suitable tracing mode; the documented retain-on-failure option records tests and retains failed runs. ^[2]

Compare the failure point with the last successful observation. Did the address change before submission? Did the API return the intended value? Did the UI render it before the assertion ran? Those questions distinguish data interference, application behavior and an assertion that observes too early.

An agent’s first deliverable should be a short failure timeline with linked artifacts and competing explanations. It should not begin with a broad rewrite of the test helper.

Change one condition at a time

Run the failing test alone against a clean fixture. Then restore the surrounding tests and original concurrency. Keep the application revision fixed while comparing results. A pass in isolation points toward a dependency worth investigating; it does not prove that concurrency is the cause.

The table is NO FEED’s diagnostic sequence. None of these symptoms establishes a cause by itself. A clean isolated run can be a useful negative observation without resolving a rarely occurring fault.

Playwright discards a failed worker process and browser, then starts a new worker; retries occur in that new process and relevant setup hooks can run again. A passing retry is therefore not necessarily the same environment simply given more time. ^[1]

For the checkout example, assign a distinct test account to each concurrent worker and repeat the original workload. Record failures and attempts before and after, retaining the assertion about the delivery address. If the failure disappears only after changing data ownership, the result supports that explanation; it still needs enough repeat observations to be convincing for this suite.

Identify what the runner isolates—and what your application shares

Playwright creates browser contexts that isolate cookies and browser storage. That clean browser state is useful, but a fresh context can still sign in to the same backend account as another test. Context isolation does not give your application a separate database record. ^[3]

Map the mutable resources used by the failing journey: account, cart, address, queue and external service fixture. Give each an owner or an explicit coordination rule. Our application-state isolation analysis^[4] covers the same distinction for parallel coding tasks; here it explains interference between test runs.

If the evidence instead shows that the assertion runs before the expected UI state exists, prefer an assertion that waits for that state. Playwright’s best-practices guide distinguishes awaiting expect(locator).toBeVisible() from immediately checking the boolean returned by isVisible(). The latter does not wait for the expected change. ^[5]

Adding a fixed sleep without identifying the required state turns an unknown delay into a guessed delay. In the checkout example, waiting longer cannot establish which worker owns the address. A timing fix is appropriate only when the observed mechanism is timing.

Give the agent a repair boundary

Our proposed instruction is: diagnose the failure, retain the protected user outcome, make the smallest supported change and report any reduction in coverage explicitly. Deleting the address assertion, skipping the test or replacing its expectation with the current output changes what the suite promises. Those actions need a separate product or test-design decision.

A legitimate test repair can change a brittle selector or obsolete expectation. Explain the current requirement and show how the replacement still fails when that requirement is violated. For the address check, deliberately supplying the wrong expected address in a disposable test run is one possible sensitivity check. It is not permission to modify production data.

If temporary quarantine is necessary, our proposed record names the owner, defect, affected coverage, exit condition and review date. Keep the test running in a visible non-blocking lane where feasible. An indefinitely skipped check loses both protection and evidence about whether the underlying behavior improved.

A visible quarantine still removes protection

On 15 October, u/Typical_Peace_8683 described teammates rerunning familiar flaky CI failures without inspection. In the replies, Torutofu_Raeva reported moving tests that flip on rerun into a non-blocking job with an owner and ticket. Other replies emphasized local reproducibility or argued for deletion. ^[6]

The discussion supplies a practical tradeoff, not a proven repair: quarantine makes the defect visible while removing its ability to block a merge. No traces or quantified improvement were provided, and the report does not attribute the flakiness to coding agents. Use that distinction when an agent proposes quarantine as the fix.

Count attempts, not just green builds

Keep retries bounded and report first-attempt failures separately from final outcomes. Compare the repair under the conditions that exposed the problem, including concurrency and fixture setup. A series of passes with no explanation is weaker than passes accompanied by an observed mechanism and an unchanged assertion.

Before the next automated CI repair, give the agent one failed attempt and a requirement it may not weaken. Ask it to identify the experiment that would distinguish its leading explanation from the closest alternative before it edits the test.