Twice the bill, the same answer

01What we told you on Wednesday

On 16 September this desk printed a take on a preprint filed the day before. The line was that on SWE-bench Verified the harness you wrap a model in moves the resolution rate by up to 29.8 points, against an 8.8-point spread across the entire top thirty, so a rank won by swapping scaffolds is a fact about your plumbing.¹²

The figure is in the abstract and we read it correctly. The framing around it was lazy. Those 29.8 points are the observed range of within-model scaffold scores across 254 published submissions that nobody controlled — different task subsets, different effort settings, different submission dates, different everything. The authors say as much in the same abstract: the observational design "does not identify causal scaffold effects".¹

A reader who took Wednesday's item as advice would go shopping for a harness that buys them points. The first controlled measurement this desk has been able to open says those points are largely absent, and that something else is moving instead.

02Fourteen comparisons: one score, eight bills

HarnessTax was published on 16 September by six authors at UC Berkeley and Arena. It runs seven models through three harnesses — Claude Code, Codex CLI and Pi — on SWE-bench Lite and Terminal-Bench 2.0: thirty randomly sampled tasks per benchmark, three attempts each, every harness in its native configuration at its high-effort setting, each attempt capped at 100 agent turns. Token cost is computed from a fixed direct-API price list dated 1 September 2026, with the same prices applied to a given model across all three harnesses.³

That pricing detail is what makes the comparison readable at all. When two harnesses running the same model on the same task produce different costs here, the difference is in tokens consumed, not in what a lab charges for them.

The SWE-bench Lite results ship as machine-readable JSON, so the paired comparisons can be checked line by line. There are fourteen of them, each against Pi as baseline, each holding the model and the thirty tasks fixed. The success-rate difference survives Holm correction in exactly one: Claude Opus 4.8 under Codex, +6.7 points over Pi, 95% CI [2.2, 11.1], p=0.002, Holm-adjusted 0.042. The cost difference survives in eight.⁴

Claude Fable 5 under Claude Code scores 97.8% against 96.7% under Pi — one task went each way and 28 landed identically — while costing 2.00x as much, 95% CI [1.76, 2.33], on 29 of the 30 tasks.⁴ The widest gap is GPT-5.6 Luna: 5.09x under Claude Code against Pi, CI [4.47, 5.80], more expensive on 30 tasks out of 30, for a success rate of 55.6% against 53.3% at p=0.734.⁴

The authors report the same shape as geometric means across shared models: Claude Code costs about 2.0x Pi and 1.6x Codex on SWE-bench Lite, and 1.5x Pi on Terminal-Bench 2.0, while the average harness effect on success rate stays within ±2% and ±5% respectively.³

The harness moved the invoice eight times as often as it moved the outcome.

03Where the money goes

The extra spend does not come from extra work. For Fable 5 on SWE-bench Lite, Pi averages 15.4 turns per attempt and Claude Code 15.3, at roughly double the cost — so the difference is per recorded turn, on turn definitions that each harness sets for itself.³

The authors measured the first main model call across all seven models and found Claude Code's mean initial context runs over 10x Pi's, with longer instructions and larger tool schemas.³ Pi is a minimal harness that exposes four tools: read, write, edit and bash.³⁵

The authors are careful about what that explains, and we will be too. A larger first call can raise cost; total spend also depends on caching, generated tokens and later calls in the run, and nobody has decomposed the 2.0x into those parts.³ What has been measured is that the ratio holds on 29 of 30 tasks, which makes it a property of the harness rather than of one unlucky repository.

04The axis you are already fighting on

Vercel published its September Production Index on 17 September, covering gateway traffic through August. Open-weight models ran 56% of tokens for the first time while accounting for 14% of spend; the average price per token fell 23.2% in August, a third consecutive monthly drop; and among teams running more than ten million tokens in both months, the median team paid 7.6% less.⁶

That 7.6% is what a month of deliberate model-shopping bought the median team on one gateway — one gateway's customers, and the operator sells the routing that makes switching cheap, so read it as a fact about that population. Set it beside a 2.0x standing difference between two harnesses running the same model on the same thirty tasks, and the effort is going into the smaller number.

Stepping down a model tier is the move teams reach for, and on this data it can be the expensive way to economise. Claude Haiku 4.5 under Claude Code costs $0.426 per rollout at a 52.2% success rate. The same model under Pi costs $0.374 at 60.0% — $0.623 per solve against $0.816.⁷ The rate gap between those two rows does not survive correction (p=0.049, adjusted 0.935), so the defensible claim is narrower and still awkward: cheaper, with no measurable loss in success.⁴ HarnessTax lists Haiku under Claude Code as dominated by five other systems, among them GPT-5.6 Luna under Pi at $0.030 per rollout and 53.3%.⁷

05What this does not show

Thirty tasks and three repetitions is a small study, and the confidence intervals say so — most success-rate intervals here are fifteen points wide. A design this size can detect a large accuracy effect and would miss a five-point one. The single comparison that did survive correction is evidence that harnesses can move accuracy; thirteen others are evidence that we cannot currently say when.

Both benchmarks are open, and the authors note the models may have encountered them during training.³ For SWE-bench Lite the authors blocked external network access and disabled the default web tools in Claude Code and Codex, which measures a richer harness with part of its equipment switched off.³ Two packages were added to Pi to configure subscription keys and control agent turns.³

On funding: Arena sponsored API access for the profiling runs and one of the six authors is at Arena; Anthropic credits came through the Laude Institute and compute credits through an Amazon fellowship.³ Pi is a third-party open-source project, not the authors' own harness.³⁵ None of that is disqualifying and all of it belongs in the margin of a cost comparison between commercial harnesses.

One boundary matters more than the rest for most readers. The price list is direct-API. On a subscription seat, a harness with a 2x token appetite would show up as quota rather than as an invoice — a mechanism, and one nobody has measured. The reports across r/ClaudeCode and r/codex of weekly-limit drops of 30% to 60% since 14 September are reader-posted and unverified; we said so on Thursday and they are still unconfirmed on any vendor page.⁸