the short version
- In FrontierHarness Eval v1.1, 64.5% of list-price inference cost went to trials that did not pass: 52.7% to failures and 11.8% to timeouts.
- Even the five best-value harnesses spent 60% (Codex CLI) to 69% (Alma) of their cost on trials that did not pass.
- A wall-clock cap acts mainly on the timeout slice: 68% to 82% of failed runs with the five best-value harnesses ended on a final answer, not at the time limit.
- Steps per pass correlated with cost per pass at +0.51 across 22 harnesses. Prompt size, tool count and cache hit rate correlated at −0.03 to +0.02.
Runta's FrontierHarness Eval v1.1 ran 27,280 coding agent trials. 64.5% of the list-price inference cost went to trials that did not pass: 52.7% to failures and 11.8% to timeouts. No harness avoided this. Even the five best-value harnesses spent 60% (Codex CLI) to 69% (Alma) of their cost on trials that did not pass. With those five, 68% to 82% of failed trials ended on a final answer rather than at the time limit, so a tighter time limit would not have stopped most of them.
Runta built the benchmark with Baseten. It covers 22 harness configurations, 5 open models, 62 terminal tasks and 4 attempts per task, for 83B tokens in total. That is 76 times the size of v1.0, which tested one model, Kimi K3. Each harness ran in Harbor with default settings at maximum reasoning effort. Every trial got a fresh Runta runtime that was deleted afterwards, and all costs are estimates at Baseten list prices. On Hacker News, a commenter who identified as Runta's founder noted that the eval runs on Runta's own runtime and told readers to weigh it accordingly.
Where the FrontierHarness Eval inference spend went
The post names a best-value harness for each model, defined by the lowest cost per pass with a tolerance rule. The five are Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse, and Alma with Kimi-K3. For these five, trials that did not pass took 60% (Codex CLI) to 69% (Alma) of each harness's cost. The post's text does not give the failed-run share for the other 17 harness configurations. That breakdown appears only in Figure 8.
Across all trials, cached input made up 42% of cost, output 33% and uncached input 25%. The average trial used 2.0 to 4.8 million input tokens, depending on the harness. Every step resends the context, so a failed run that iterates for a long time keeps paying for its growing context, even when most of it is cached. Cached input costs 2% (DeepSeek) to 20% (GLM Flash) of the uncached price, which affects how expensive a long failed run is on each model.
Failed runs end confidently, not at the time limit
Among the five best-value harnesses, 68% (Pi) to 82% (Codex CLI) of failed trials ended with a final answer rather than at the time limit. Runta says few of those answers stated that the work was incomplete. It published one Codex CLI trace on GLM-5.3, in which the agent reported success and the verifier failed the task:
# Codex CLI, GLM-5.3, cancel-async-tasks (trial 37182b0e)
Agent: - Validated with system Python, including concurrency and
cancellation-cleanup tests.
Verifier: FAILED test_tasks_cancel_above_max_concurrent
assert stdout.count("Cleaned up.") == 2
AssertionError: assert 0 == 2This limits what a cap can do. Task time limits ranged from 15 minutes to 8 hours, and a tighter wall-clock limit acts directly on the 11.8% spent on timeouts. A token or step cap could also shorten failures that run long before the agent stops on its own. It would not catch failures that end early with a wrong answer. The post does not report how many steps or tokens failed runs used compared with passed runs, so this data cannot set a cap threshold. Runta suggests that a check before the harness stops could save part of the failure spend, but it says it did not test this.
Steps, not prompt size, predict cost per pass
Runta compared four harness factors against cost per pass: first-request tokens, tool count, cache hit rate and steps. Across the 22 harnesses, steps per pass had a rank correlation of +0.51 with cost per pass (p = 0.015). The other three factors ranged from −0.03 to +0.02 (p ≥ 0.88). mini-SWE-agent has the smallest first request, at 882 tokens, but it took 38 steps per pass. It ranked 17th of 22 on cost per pass and 19th of 22 on cost per task. The largest first request, 21,365 tokens, is less than 1% of the 2.0 to 4.8 million input tokens an average trial used.
One commenter on Hacker News called the steps finding close to tautological, since cost scales with tokens and tokens scale with steps. The commenter also argued that a harness does not control its step count directly, the way it controls its prompt and tools. The more useful part of the result is the null finding: neither the system prompt size nor the tool count predicted cheaper passes. The post adds that tool calls tracked cost as well as steps within each model. It does not test whether a step limit cuts cost without cutting pass rate. The founder's comment on Hacker News says output tokens predict cost, while the post itself names steps. This note follows the post.
The harness changes cost 1.8 to 5.0 times per model
The choice of model explains 92.8% of the variance in log cost per task. The consistent harness effect explains 2.3%, and the harness-model pairing 4.9%, which includes noise. With the model fixed, 68% of the variation between harnesses depends on the pairing or is noise. The most expensive harness cost 1.8 to 5.0 times as much per task as the cheapest. The largest gap was on DeepSeek, where Cline CLI cost 5.0 times as much as OpenCode v1 for 2.0 more points of pass rate.
Cost rankings move between models. Codex CLI was 1st on cost per task with GLM-5.3 and 16th with Kimi-K3. Spending more did not buy passes. The harness with the highest pass rate ranked 5th to 20th of 22 on cost per task. Within each model, the rank correlation of cost per task and pass rate ranged from −0.39 to +0.22.
Another commenter doubted that the conclusions would hold up under a second sample, given the variability in the results and the large share of cost from failed runs. Runta reports three checks:
- A permutation test shuffled trials between harnesses 300 times. All five real cost ratios were larger than every shuffle (p = 0.003).
- Cost rankings from two halves of the attempts had a mean rank correlation of 0.71 within a model and 0.26 across models.
- Across 4,000 task resamples, Codex CLI stayed the best value with GLM-5.3 in 99.6% of draws. The other four winners were picked in 25% to 53% of draws.
Cost rankings look stable within a model. Pass rates are less certain: each has a 95% interval of about ±10 points, so a gap of a few points is a tie. Some figures in the founder's comment differ from the post: 4.4x, Codex CLI ranking 15th with Kimi-K3, and a range of 4th to 19th. This note uses the post's figures.
Budgeting runs against a 64.5% failure share
The data supports some budgeting practices and leaves others untested. These are the points an engineer can act on:
- Budget on cost per pass, not cost per task. Runta's best-value rule uses cost per pass, and the two metrics can rank a harness differently: mini-SWE-agent was 17th on one and 19th on the other.
- Re-run harness selection on your own tasks whenever you change the model. Runta recommends a private dataset large enough to see past the noise, and pass rates at 62 tasks still carried about ±10 points.
- Track steps per trial as the main cost signal. Prompt size and tool count did not track cost per pass in this data.
- Treat a confident final answer as unverified until your own tests run. In this benchmark, most failures ended with a final answer.
- Expect timeout caps to act mainly on the 11.8% timeout share. A verification check before the agent stops is the lever Runta points to for the 52.7% failure share, but Runta says it did not test it.
Separately, audits flagged 216 trials for reward hacking, 163 of them online lookups, and those trials scored 0. The results cover open models served by one provider, Baseten. Runta notes that closed models, or another provider's caching and prices, could change the cost ranks. Its next step is to add closed models such as Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon.
Questions this raises
How much of the FrontierHarness Eval v1.1 spend went to failed runs?
64.5% of the list-price inference cost went to trials that did not pass, with 52.7% on failures and 11.8% on timeouts. Even the five best-value harnesses spent 60% (Codex CLI) to 69% (Alma) of their cost on trials that did not pass.
Would a tighter time limit cut the cost of failed coding agent runs?
Only partly. A tighter wall-clock limit acts directly on the 11.8% spent on timeouts, but among the best-value harnesses 68% to 82% of failed trials ended on a final answer before the limit. Runta suggests a check before the harness stops could save part of the failure spend, but it did not test this.
What predicts cost per pass for a coding agent harness?
Steps per pass had a rank correlation of +0.51 with cost per pass across 22 harnesses. First-request tokens, tool count and cache hit rate showed no meaningful correlation. mini-SWE-agent had the smallest first request but took 38 steps per pass and ranked 17th of 22 on cost per pass.
Spotted an error? Email samir.s.sengupta@gmail.com and I will correct it and date the change at the top of the note. How these notes are researched, drafted and checked is in the editorial policy.
