# HarnessTax: the same coding model, similar success, up to 5x the cost across harnesses

> A Berkeley study ran seven models in Claude Code, Codex CLI and Pi: the harness barely changed success rates but moved cost by up to 5x, and the minimal Pi stayed competitive.

- Published: 2026-09-17 (updated 2026-10-17)
- By: Samir Sengupta (https://www.samcodeman.com/about)
- Canonical: https://www.samcodeman.com/writing/harnesstax-coding-agent-harness-benchmark
- How this was made: Drafted with Claude, Anthropic's model from 1 cited source, then checked against it in a second editing pass. Editorial policy: https://www.samcodeman.com/editorial-policy
- Disclosure: I build Wayne, an AI coding harness for VS Code, and Omnitron, an agent harness, so I work in this category.
- Correction (2026-10-17): An earlier version of this note said the HarnessTax page published only a title and no results. The page loads its results with JavaScript; it reports 21 model-harness pairs across two benchmarks. The note has been rewritten from the full study.

## The short version

- Across 21 model-harness pairs, the harness moved success rates by about 2% on SWE-bench Lite but moved cost by up to 5x.
- Pi, a minimal harness with four tools, reached the cost-success Pareto frontier on both benchmarks.
- An alternative harness beat the provider's own in nine of twelve comparisons for Anthropic and OpenAI models.
- Claude Code's first-call context was over 10x Pi's, which raises cost before any work is done.

HarnessTax measured 21 model-harness pairs - seven models in Claude Code, Codex CLI and Pi - on SWE-bench Lite and Terminal-Bench 2.0, and found that the harness barely moves the success rate but can multiply the cost: the same model reached similar success at up to 5x the cost. Claude Fable 5 solved 97.8% of attempts in Claude Code and 96.7% in Pi, at $1.33 versus $0.67 per attempt.

The study comes from Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei Zaharia at UC Berkeley and Arena Intelligence. Its question is narrow and practical: when you pick a coding agent, you pick a model and a harness - the software that manages the model's tools, context and task execution - and nobody had measured how much the second choice costs you.

## What HarnessTax tested and how

The seven models are Claude Fable 5, Claude Opus 4.8, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.6 Sol, GPT-5.6 Luna and the open-weight Kimi K3. Each ran in three harnesses: Anthropic's Claude Code, OpenAI's Codex CLI, and Pi, a minimal open-source harness. Every model-harness point is 30 randomly sampled tasks over 3 repetitions, on two open-source benchmarks: SWE-bench Lite and Terminal-Bench 2.0.

Cost is the mean token cost per rollout, in US dollars. The figures plot it against success rate and draw the Pareto frontier - the best success rate observed at or below each cost level - which is the right lens for a buying decision: the question is not which pairing scores highest, but what each extra point of success costs.

## Harness choice changes cost far more than success

Across models that ran in all three, Claude Code cost about 2.0x as much as Pi and 1.6x as much as Codex on SWE-bench Lite, and about 1.5x as much as Pi on Terminal-Bench 2.0, measured as geometric means of cost ratios. Over the same runs, the average harness effect on success rate stayed within plus or minus 2% on SWE-bench Lite and about 5% on Terminal-Bench 2.0.

Fable 5 shows the pattern most clearly: 97.8% in Claude Code, 96.7% in Codex and 96.7% in Pi, with Claude Code at about twice Pi's cost. The authors call that gap a harness tax, borrowing the term from a Portkey post: paying more for essentially the same result because the default harness was never compared with the alternatives.

At the ends of the frontier, GPT-5.6 Luna was the cheapest option on both benchmarks and Fable 5 reached the highest success rate on SWE-bench Lite. Kimi K3 sat close to the frontier near GPT-5.6 Sol on SWE-bench Lite and just below it on Terminal-Bench 2.0.

## Why Pi, with four tools, keeps up

Pi gives the model four tools - read, write, edit and bash - and still reaches the Pareto frontier on both benchmarks. The cost difference does not come from Pi taking fewer steps: for Fable 5 on SWE-bench Lite, Pi and Claude Code averaged 15.4 and 15.3 turns per attempt, yet Claude Code cost about twice as much for a 1.1% higher success rate, which means more spending per turn. The authors note that turn definitions differ between harnesses.

Part of the gap starts before the first tool call. Across all seven models, Claude Code's mean initial context was more than 10x Pi's, with longer instructions and larger tool schemas. The study is careful about what follows from that: total spending also depends on caching, generated tokens and later calls, so a bigger first prompt raises cost without fully explaining it.

## Models often do better outside their own harness

Across the six Anthropic and OpenAI models on both benchmarks, a harness other than the provider's own reached the highest observed success rate in nine of twelve comparisons. Sonnet 4.6 solved 68.9% of SWE-bench Lite attempts in Codex against 66.7% in Claude Code at a similar cost, and GPT-5.6 Sol reached 83.3% on Terminal-Bench 2.0 in Pi against 78.9% in Codex, at about half the cost: $0.42 versus $0.76.

That cuts against a common assumption, that a provider's harness is the best home for its own models because the two are tuned together. On these benchmarks a shared provider did not guarantee the best pairing, and the authors' summary is blunt: your Claude models may not need Claude Code.

## What it means if you run coding agents

- Measure cost per solved task, not only success rate: on these runs the success column barely moved while the cost column moved up to 5x.
- Evaluate a model in more than one harness before standardising on it; the default pairing was not the best one in nine of twelve comparisons.
- Count the first call: a harness that ships long instructions and large tool schemas pays for them on every task, and caching changes how much.
- Treat harness features as a trade-off to test, not a free upgrade; a four-tool harness was competitive here.

The authors' own limit applies to all of it: two open-source benchmarks that the models may have seen in training, so results on private codebases and other workloads may differ. The authors say they will release their profiling traces, and with 30 tasks per point, small gaps between pairings should be read against the 95% confidence intervals the charts show.

## What the study leaves open

HarnessTax does not cover real development workflows, where requirements change, developers give feedback and tasks run across sessions; the authors name that as the next step, along with automating harness selection. It also tests three harnesses, so it says nothing about the many other agent frameworks in use. What it does establish, on the two benchmarks it ran, is that the harness is a cost decision that deserves the same scrutiny as the model.

## Questions this raises

### What is HarnessTax?

HarnessTax is a study from UC Berkeley and Arena Intelligence that measures how much a coding agent's harness changes its success rate and cost. It ran seven models in Claude Code, Codex CLI and Pi on SWE-bench Lite and Terminal-Bench 2.0.

### Does Claude Code solve more tasks than Pi or Codex CLI?

Only slightly in this study. Claude Fable 5 solved 97.8% of attempts in Claude Code and 96.7% in both Codex and Pi, while Claude Code cost about twice as much as Pi. Across models, the harness changed success by about 2% on SWE-bench Lite.

### Which harness was cheapest for the same results?

Pi, the minimal four-tool harness, reached the cost-success Pareto frontier on both benchmarks, and Codex was also cheaper than Claude Code. On SWE-bench Lite, Claude Code cost about 2.0x Pi and 1.6x Codex across shared models.

## Key facts

- 21 model-harness pairs: 7 models x Claude Code, Codex CLI and Pi, on SWE-bench Lite and Terminal-Bench 2.0
- Claude Fable 5: 97.8% in Claude Code vs 96.7% in Pi, at $1.33 vs $0.67 per attempt
- Claude Code cost about 2.0x Pi and 1.6x Codex on SWE-bench Lite (geometric mean of cost ratios)
- GPT-5.6 Sol on Terminal-Bench 2.0: 83.3% in Pi vs 78.9% in Codex, $0.42 vs $0.76
- Each point: 30 randomly sampled tasks over 3 repetitions

## Sources

1. [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/) - UC Berkeley Sky Lab and Arena Intelligence