Before an AI refactor, record what the old code actually does

Consider a hypothetical invoice exporter. It reads several old database tables and produces a CSV consumed by another system. The implementation is difficult to maintain. You ask an agent to split it into smaller functions. Which details of that output must survive?

Characterization tests preserve observed behavior while you change an implementation. Build them against the old revision, inspect the captured results, and keep them fixed during the refactor. They answer whether behavior changed for the cases you exercised. They do not establish that the old behavior was correct.

That distinction gives an AI-assisted refactor a manageable boundary. The agent can replace a tangled function without also deciding which undocumented quirks deserve to disappear. Michael Feathers describes characterization as an interactive process: prepare a fixture, ask existing code a question, observe the answer, then retain that interaction as a test. The expected result comes from execution, not a plausible reading of the function. ^[1]

Decide which behavior must survive

The agent notices that cancelled invoices appear as zero-value rows. Filtering them out looks cleaner. It also sorts rows by invoice number and replaces an empty customer reference with null. All three changes could be reasonable in a new exporter. None belongs in a behavior-preserving refactor unless the downstream contract allows it.

Start with the boundary the consumer sees: exported bytes, exit status, and relevant side effects. An assertion about the number of helper functions says little about that contract. A comparison of the CSV before and after the change can reveal the altered rows, order, or empty value.

Alberto Savoia’s characterization-testing walkthrough explicitly separates current behavior from intended behavior and recommends concentrating cases around the area being changed. That is the useful scope here: capture the exporter’s observable decisions before redesigning its internals. ^[2]

Choose fixtures that make decisions visible

Our proposed workflow starts with a small, named set of synthetic inputs. Use examples derived from known production cases only after removing personal data and secrets; there is no need to copy a customer database into an agent workspace.

For the exporter, each fixture should expose a decision the refactor might accidentally change:

This table is a proposed fixture design, not a report about a real billing system. Add cases because they exercise a distinct branch or dependency, not because a large fixture count looks reassuring.

Run the old revision first. Record its commit, runtime, fixture version, command, and output. If the same case changes between runs, investigate that variability before treating the output as a reference. Otherwise, the first refactor failure will mix implementation changes with clock, locale, or environment noise.

Normalize noise without erasing the requirement

Snapshots can compare any serializable value, not only UI trees. Jest’s documentation also warns that each snapshot covers only the data supplied to that particular test. Capturing one invoice does not protect every invoice type. ^[3]

A generated timestamp may be irrelevant to the exporter’s internal refactor. A deterministic clock is often easier to understand than deleting every date-looking string. If normalization is necessary, make the transformation narrow: replace the known generated_at field while preserving invoice dates, payment deadlines, and row order.

The danger is a comparison that becomes easier to pass by hiding meaningful differences. Sorting the entire output would conceal the agent’s ordering change. Converting every empty value to the same representation would erase the customer-reference distinction. Treat the normalizer as part of the test and review its behavior explicitly.

Jest recommends deterministic, focused snapshots and reviewing them as code. Its update command can accept new output; it cannot decide whether that output should become the contract. ^[3]

Give the agent a preservation task

A January Reddit account describes the same ordering. User thewritingwallah reports spending the first four days of a Django refactor on characterization tests before changing small units. The post promotes CodeRabbit, and a commenter challenges it as a sales pitch. We have not verified the project or its claimed outcome; it is an example of a reported workflow, not evidence of a speedup. ^[4]

Use a bounded assignment: extract the formatter from the database orchestration, preserve the approved fixtures, and report any observed mismatch. Keep the baseline-creation change separate from the implementation change so a reviewer can see which expectations existed before the agent edited the exporter.

Here is a proposed task statement:

Refactor the invoice exporter without changing the recorded external behavior. The approved fixtures and expected outputs define the preservation boundary for this task. Do not regenerate expected outputs to make failures pass. Explain each mismatch, including whether it comes from nondeterminism, a deliberate requirement change, or the implementation.

That instruction is a working agreement, not a technical access control. A stronger evaluation can run an untouched copy of the accepted tests outside the agent’s editable checkout. Ordinary code review should still examine changes to fixtures, serialization, and test setup.

Keep each refactoring step small enough to explain. Extracting one computation and running the relevant fixture set gives a failure a narrower cause than rewriting database access, business rules, and formatting together. This is NO FEED’s proposed implementation sequence; we have not measured a universal improvement from a particular patch size.

A discovered bug needs a separate decision

Suppose the old exporter rounds a fractional amount in a way the business considers wrong. Preserving that result indefinitely would be a mistake. Silently changing the expected file during the refactor would also hide a product decision.

Retain the observed behavior as evidence, then specify the desired result independently. The bug-fix change should explain why the expected output changes and which consumers are affected. Run the remaining characterization cases to check that correcting rounding did not also remove cancelled invoices.

For a high-impact defect, the correct action may be to pause the structural refactor and fix the behavior first. Characterization does not require preserving a known vulnerability or a harmful financial error. It requires making the difference visible and assigning an owner to the decision.

Use the baseline for the job it can do

An unchanged output on selected fixtures is evidence of preservation on those fixtures. It does not establish complete equivalence, adequate performance, or correct behavior on inputs you never exercised. Add a targeted check for any changed property the snapshots cannot see, such as an extra database write.

Keep the change understandable too: our decoder and code-comprehension case^[5] separates an external reference from the maintainer’s ability to explain the patch.

Before your next AI refactor, choose one consumer-visible boundary and capture three distinct decisions from the old code. Review those expected results before the agent begins changing the implementation. If nobody can explain whether a mismatch is acceptable, resolve that requirement before approving a new baseline.