The harness is a pricing decision

A developer choosing an agent also chooses how that agent spends. A September comparison kept the model fixed and found a large cost gap. Its evidence is useful; its limits matter before a team switches tools.

The argument

The harness belongs in a task budget: matched September runs show that changing it can move cost substantially, while leaving the value of the resulting work for a team to establish.

01One model, two ways to spend

HarnessTax’s September SWE-bench Lite dataset compares seven models through Claude Code, Codex and Pi. It contains fourteen matched comparisons against Pi, using thirty unique tasks with three repetitions. The model stays fixed within each comparison; the client carrying the task changes. This is a way to investigate a purchase that a model price list leaves unresolved. ²

One result is striking: with GPT-5.6 Luna, Claude Code averaged $0.15246 per rollout and Pi $0.029976, a ratio of 5.09. Reported success was 55.6% and 53.3%. These are benchmark outcomes, with the uncertainties discussed below, rather than a quotation for another repository. ² ³

Consider what that means for a team choosing an agent for billing fixes. Knowing the token price leaves open how the client opens files, calls tools and carries the conversation through. In this hypothetical trial, the team needs the cost of a complete attempt and the work that survives its own acceptance checks. The benchmark identifies a comparison worth making; it does not supply the team’s result.

Historical HarnessTax comparison: the same thirty SWE-bench Lite tasks and GPT-5.6 Luna run through Pi and Claude Code. Reported average costs differ by 5.09 times; success rates are 53.3 and 55.6 percent, without a statistically separated success difference. — The September matched comparison separates the client from the model choice. These are benchmark rollouts; the data does not establish equal success or a saving on another repository. ² ³Illustration: NO FEEDSources: HarnessTax — SWE-bench Lite harness-effect data: 14 paired comparisons against Pi, per-pair rate and cost statistics read 2026-10-18 · HarnessTax — SWE-bench Lite performance-cost frontier: cost per rollout and success rate for all 21 configurations read 2026-10-18

02What fourteen comparisons actually establish

Eight of the fourteen cost differences clear the publisher’s Holm correction for multiple comparisons. One success-rate difference clears it: Opus 4.8 under Codex, 6.7 percentage points above Pi. ²

That count describes this sample. It is not a probability that a future harness change will affect cost rather than quality. A comparison that fails to separate success rates also has not proved the clients equivalent. For Luna, the reported cost-ratio interval is 4.47–5.80; the success difference remains uncertain. The team can take the cost result seriously while still investigating the output. ²

The table keeps the original September figures. Its variation matters as much as the largest multiplier: the Sonnet and Haiku comparisons do not support the same cost claim as Luna. Selecting the most dramatic row and applying it to every model would give the team a budget with the wrong baseline.

Model	Claude Code vs Pi	Codex vs Pi	Cost clears Holm	Rate clears Holm
GPT-5.6 Luna	5.09x	1.18x	Claude Code only	neither
GPT-5.6 Sol	3.50x	1.27x	both	neither
Claude Opus 4.8	2.06x	1.47x	Claude Code only	Codex only
Claude Fable 5	2.00x	1.34x	both	neither
Kimi K3	1.72x	1.86x	both	neither
Claude Haiku 4.5	1.14x	1.05x	neither	neither
Claude Sonnet 4.6	0.99x	1.10x	neither	neither

Cost per rollout relative to the same model under Pi, SWE-bench Lite, 30 tasks over 3 repetitions. Bold-equivalent cases are those clearing Holm correction. Read 19 September 2026.²³

03A ratio leaves the cause open

The same nominal model can receive different instructions, tool definitions and conversation histories through different clients. A retry policy can buy another attempt; a context policy can send earlier material again. These are plausible ways for a harness to change spend. The published comparison does not provide the token accounting needed to assign its ratios to one of them.

For the billing team, that missing mechanism affects the next decision. If an optional retry setting accounts for the extra spend, it may be able to change that setting and keep its existing integrations. If the difference follows from an indispensable workflow, the extra cost has to earn its place through the resulting work. Neither explanation follows from the model’s price list.

The second benchmark is a useful limit. HarnessTax labels its six matched Terminal-Bench 2 pairs descriptive, without confirmatory tests. Their cost ratios run from about 1.05 to 1.81. A smaller range on another task set is a warning against carrying the SWE-bench maximum into a different workload. ⁴

04The patch still has to be accepted

Imagine both clients returning a billing patch. One has lower recorded API spend, while the other includes a failing test that exposes an ambiguous requirement. A benchmark’s success label and the team’s acceptance decision can diverge. To compare the tools, the team needs a fixed requirement, the same starting revision and a record of what the reviewer accepted, including any recovery work.

A September preprint offers a related caution about reading scores. Its audit of 254 SWE-bench submissions found observed within-model scaffold ranges up to 29.8 percentage points, compared with an 8.8-point spread across Verified’s top thirty. The authors describe an observational design that cannot identify causal scaffold effects. Published scores belong to evaluated model-scaffold pairs. ⁶

For the team’s proposed trial, completed work is the denominator. Failed attempts, retries and human corrections belong in the cost of producing an accepted patch. Startup and review time can change whether a lower model bill is useful. This is a measurement design we recommend, not a trial we have run or a claim that one client wins on private code.

05Know whose evidence you are buying

HarnessTax supplies public data, but we could not establish the publisher’s identity from its organisation metadata. At this reading the GitHub organisation gives no name, email, homepage or description. Pi is the baseline throughout. That does not invalidate the arithmetic; it limits how confidently a team should adopt the publisher’s framing without independent replication. ⁵

Thirty unique tasks with repeated runs remain thirty tasks. They do not cover the billing service’s dependencies, review standards or failure modes. Repetition helps describe variation within the selected sample; it does not turn the sample into a private monorepo.

Operating the client adds another cost that these tables do not price. GitHub’s incident record for 17 September attributes Copilot’s elevated OpenAI-model errors to an upstream provider, with a stated 20:26–21:17 UTC impact window. The delivery path and its status record matter even when the model is purchased through a client. This incident does not establish the cause of any benchmark cost gap. ⁸

→So what

Compare a harness and model together on a small set of your own accepted tasks. Keep the starting revision, requirements and acceptance criteria fixed.
Retain the complete cost record, including unsuccessful attempts and review. Investigate the largest gap before treating it as an unavoidable fee.
Use the September table to choose comparisons worth making. The 5.09x row is evidence for a local evaluation, not a migration instruction for every model.

?What would change our mind

Independent, named replications on broader task sets could show whether the large cost ratios survive outside this thirty-task sample.
Token and retry accounting could explain a gap through a user-controlled setting. That would support changing the configuration before changing the client.
A comparison of accepted billing work, including review and recovery, could justify the more expensive path. Lower rollout spend alone would then be an incomplete decision.

Sources, with the date each was read