NO FEED · AI-assisted development
How to test AI-written code without copying its assumptions
Test AI-written code against independent expectations. Compare worked examples, properties, metamorphic checks and mutation testing with one interval example.
A passing test is useful only when its expected result has a defensible basis. For AI-written code, derive the important expectations from the requirement, an independent reference, or a justified property. Asking another agent to read the implementation and produce matching tests does not, by itself, provide that independence.
The practical question is simple: if the implementation is wrong, what would make this test disagree with it? A second model can suggest cases, but it still needs a reason for each expected answer. This article proposes a review method for that reason—the test oracle—rather than a rule about which model should write tests.
Write the example before reading the answer
Take a hypothetical function that merges overlapping closed intervals. The contract says the result must preserve every covered point, contain no overlapping intervals, and be ordered by start. Intervals that touch must merge. Inputs use finite integer endpoints, with start no greater than end.
For [[0,10], [2,3], [20,25]], the required result is [[0,10], [20,25]]. The small interval sits inside the first one. A faulty implementation might overwrite the current end with 3, producing [[0,3], [20,25]] and losing covered points.
A test generator that runs the faulty function and copies its output into an assertion could preserve that mistake. Recording current output is appropriate when intentionally characterizing legacy behavior. It is inadequate when the task is to establish whether a new algorithm satisfies its contract.
Our proposed first step is to write one worked example without calling the implementation. Explain the expected result in a sentence. For this example: points from four through ten were covered in the input, so they must remain covered in the output. The explanation makes the assertion reviewable even if the code changes completely.
Match each test to a different reason for failure
Different checks expose different weaknesses. More tests of the same ordinary input may add less information than one nested interval or a deliberately altered comparison.
| Check | Independent basis | What it can expose | What it cannot establish alone |
|---|---|---|---|
| Worked acceptance example | Explicit contract and manual derivation | A known business or boundary case | Behavior outside those examples |
| Differential comparison | Trusted alternative implementation | A disagreement on the same valid input | Which implementation is correct |
| Property-based test | A justified invariant over a defined input domain | Broad families of counterexamples | Whether the property is sufficient |
| Metamorphic test | A justified relation between transformed inputs | Inconsistent behavior without a full expected output | All errors that preserve that relation |
| Mutation test | Deliberately changed implementation | A test suite that misses that modeled fault | Absence of other real defects |
This is a proposed selection aid. It does not mean every pull request needs five testing systems. Choose the method that challenges the most consequential uncertainty.
Properties need a precise domain
Hypothesis lets a test describe a range of inputs through strategies and evaluates examples from that range. Its documentation shows both generated inputs and counterexamples; it does not turn an incorrect assertion into a correct requirement. [1]
For the interval function, generate integer endpoints within small, fixed bounds. Require output intervals to have integer endpoints within those bounds, with start no greater than end. Compare input and output coverage at every integer and half-integer in the range. Integer probes check closed boundaries; midpoint probes check the spaces between them. With integer endpoints, coverage cannot change inside those spaces.
Also check that the output is ordered and that adjacent output intervals no longer overlap or touch. Preserving coverage alone would allow the function to return the original, unmerged list. An assertion must exclude the wrong implementations you actually care about.
Integer probes alone miss gaps: [[0,5],[6,10]] covers every integer in [0,10] but loses the space between five and six. The midpoint 5.5 catches that error. This bounded oracle is unsuitable for arbitrary floating-point endpoints or enormous ranges; those need separately justified checks.
Relations can reveal mistakes without a full answer
Metamorphic testing checks relationships between executions when computing a complete expected result is difficult. The original research proposes deriving further test cases through such relationships rather than treating successful cases as finished work. [2]
For our stated interval contract, permuting the input must not change the merged result. Replacing [0,10] with [0,5] and [5,10] must preserve it too. These transformations are justified by the contract’s coverage and touching rules.
An idempotence check—merging an already merged output changes nothing—is useful but weak on its own. A function that always returns an empty list also satisfies it. Pair relations with at least one concrete expected result and checks that preserve the input’s information.
The same caution applies to round trips. If an encoder and decoder share the same mistake, their composition can succeed while disagreeing with the external format. A published specification or independently produced fixture is what connects that round trip to something outside the two implementations.
Deliberately break the behavior you depend on
Reddit user ArgumentAcrobatic250 describes an archive test that stayed green after deliberately changing production to copy only the first page. A stricter replacement then rejected correct behavior. In the author’s account, the useful test finally passed normal code and rejected the intentional break. Replies disclose the model settings but also that exact prompts were not retained. This is a self-report without an independently checked replay, not a general verdict on AI-generated tests. [3]
Mutation testing changes program expressions and observes whether the tests detect the change. Stryker distinguishes killed mutants from survivors and also records states such as no coverage and timeouts. Read those outcomes rather than treating one percentage as a complete judgment. [4]
For the interval example, replacing the maximum end calculation with a minimum should fail the nested-interval case. Changing the touching comparison should fail a fixture that makes the contract explicit. If those tests remain green, inspect whether the altered code was executed and whether the assertion checks the lost behavior.
Do not demand a perfect score mechanically. Some mutations are equivalent under the program’s semantics and cannot be distinguished by a valid input; Stryker documents that limitation. A surviving mutant needs interpretation before it becomes a new test requirement. [5]
For a small change, a manually introduced fault on a disposable branch may answer the immediate question. For repeated work on a critical module, an automated mutation tool can make that check repeatable. Neither route proves the module has no bugs.
Review the expectation separately from the patch
Before accepting an AI-generated test, write down where its expected value came from. If the answer is “the function returned it,” decide whether this is intentional characterization or accidental circularity. If the answer is “another model agreed,” find the requirement or reference that agreement depends on.
Our decoder case[6] shows a developer using a reference implementation. Being able to explain the resulting patch remains a separate acceptance question.
For your next change, select one assertion that protects the main user-visible outcome. Explain its expected value without reading the implementation, then introduce one plausible wrong behavior and confirm the check rejects it. Keep that explanation with the test so the next agent cannot silently redefine success.
Sources & context
- Hypothesis — QuickstartOfficial documentation; read 19 October 2026.
- Chen, Cheung and Yiu — Metamorphic Testing: A New Approach for Generating Next Test CasesOriginal research report, archived on arXiv in 2020; read 19 October 2026.
- ArgumentAcrobatic250 — Testing AI-written archive testsReddit, 24 August 2026; original and 11 comments checked on 19 October. Outcome not reproduced.
- Stryker — Mutant states and metricsOfficial documentation; read 19 October 2026.
- Stryker — Equivalent mutantsOfficial documentation; read 19 October 2026.
- decoder caseRead 19 October 2026.