Only 28 of 520 agent runs actually completed a whole-repo stack migration

SWE Refactor Bench, posted to arXiv on 24 August 2026, evaluates whether coding agents can perform long-horizon, whole-repository stack migrations, and it checks that the migration actually occurred rather than only that the tests still pass. Across 520 runs from 8 frontier models and 26 model-effort configurations, 28 runs (5.4 percent) passed all three evaluation stages, and 13 of the 20 tasks received no accepted solution at all. The gap between those two numbers and the pass rates people quote from bug-fixing benchmarks is the whole point of the paper.

The setup is 20 whole-repository migrations covering 4 kinds of technical debt. The abstract does not name which four, and it does not describe the repositories, so treat the task distribution as unknown until the full paper is read. What is stated clearly is the motivation: modern software systems accumulate technical debt over decades of development, migration is expensive and largely manual, and existing benchmarks cannot tell you whether an agent has done the work because they evaluate only behavioural correctness.

If your only signal is a fixed test suite, an agent has a cheap way to win. It can keep the original implementation in place, wrap or copy it, and let the suite go green without ever migrating anything. The authors give this failure mode a name.

Blindness is not an exotic adversarial case. It is the locally optimal move for any agent whose reward is defined by a test command, and it is exactly the behaviour a migration task invites, because the old implementation is sitting right there in the repository. Any internal eval that scores migrations on test outcomes alone is measuring something other than migration.

SWE Refactor Bench separates completeness from correctness and then goes looking for what both missed. Stage one, Migration Audit, verifies that the migration occurred. Stage two, Behavioural Tests, measures correctness against a fixed test suite. Stage three, Agentic Verification, uses 6 independent coding agents to generate targeted tests for hidden behavioural differences.

The third stage is the one worth copying. A fixed suite only covers behaviour someone already thought to encode; a migration that changes a data access layer or an async model can preserve every asserted behaviour while silently changing an unasserted one. Using independent agents to write adversarial tests against the diff turns the eval into a search for undeclared behaviour change rather than a checkbox. It also means an accepted solution has to survive tests it was not trained or prompted against.

The headline result - 5.4 percent of runs accepted, 13 of 20 tasks with no accepted solution at all - says the failures cluster in tasks rather than distributing evenly, which means an aggregate pass rate will hide where your agent breaks. Score per task, and report the audit stage separately from the correctness stage, because they fail for different reasons. A run that passes the audit and fails behavioural tests is an agent that migrated and broke something. A run that fails the audit and passes the tests is Blindness, and it should be scored as a zero, not a partial credit.

When a task fails, the next question is which layer to fix, and Joe Wright's terminology post is a useful discipline here: the agent system is a harness, an inference service and a model, and the same model behaves differently across harnesses because the harness shapes the inputs and interprets the outputs. Missing context is a harness problem. A malformed tool call is a harness or model problem. Bad reasoning is the model or the context the harness supplied. SWE Refactor Bench reports 26 model-effort configurations, which suggests effort level is a variable worth sweeping before you conclude a model cannot do the task.

Harness design is an open variable for long-horizon work specifically. Laude's Headlong, released 25 August 2026, is an open source microharness with a core of less than 10K lines of Bash in which the agent keeps generating thoughts in a self-guided loop rather than sitting frozen between requests. It is alpha research software aimed at persistent agency rather than repository migration, and no migration benchmark results are reported for it, but it is a concrete example of harness-level variation that a per-task migration score could be used to evaluate.

The abstract names claude-opus-5 as the best model but the text is truncated before its score, so the per-model spread is not available from the abstract alone. The four kinds of technical debt are not enumerated, the repositories are not identified, and there is no reported cost, wall-clock time or token budget per run - all of which matter if you are deciding whether an autonomous migration is cheaper than the manual one it replaces. The paper also does not say how often Blindness specifically accounted for a rejection versus ordinary incompleteness.

One adjacent thread to watch: Best-Practice Critic Optimization, posted the same day, conditions a training-time critic on reward-defining information such as a reference answer or grading rubric that is hidden from the policy, and reports that the recipe also improves learning with rubric-based rewards. A migration audit is a rubric. Whether that kind of hidden-rubric supervision transfers from mathematical reasoning to whole-repository migration is untested in either paper, and until someone tests it, the practical advice stands on the eval side: audit the migration, then test the behaviour, then let independent agents attack the diff.