Real-SWE benchmark private enterprise codebases: 38.8% top resolve rate

Real-SWE does not publish resolve rates for the same models on SWE-bench-style public benchmarks, so there is no supported head-to-head shift to quote. The only cross-benchmark numbers it gives are median prompt length and median files edited by the reference solution, both covered below. Any claim about how far rankings move relative to a public benchmark would have to come from somewhere other than this release.

The eighth-place entry, GPT-5.6 Sol in Codex CLI, lands at 16.2%, less than half the top score. Between the ends sit GLM 5.3 in Claude Code at 28.8%, Grok 4.6 in Grok Build and Muse Spark 1.3 in Muse Code tied at 23.8%, and Kimi K3 in Kimi Code at 18.8%. Every number is a model-and-harness pair: Specific Labs says it uses native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.

Each task comes from a private production codebase licensed from a real-world company, and the tasks are problems that company's engineers work on. Codebases were screened for companies with substantial usage, strong engineering teams and demanding production workloads. The examples disclosed are a Luma/Partiful competitor with 200K+ users and a top 100 App Store ranking, a consumer fintech platform processing 100K+ bank statements, and enterprise AI sales platforms supporting complex business workflows. Specific Labs says it prioritises code written to meet an actual user or business need over code written solely to create a benchmark task.

The contamination argument is that tasks on private codebases are natively out of distribution, because the code and its solutions are not available anywhere on the public internet, and the post states that 99% of tokens in real-world enterprises are hidden from frontier models. Every task is described as inspired or lifted verbatim from a private, real-world codebase. The release reports no contamination measurement against the evaluated models; the out-of-distribution property is asserted from provenance, not tested.

Tasks carry business consequences: getting billing right, calculating taxes, migrating customers, often across multiple services. The published tax example asks an agent to fix invoice billing in a NestJS TypeScript service so each business charges the right tax and exempt customers are not taxed, routes destination pricing to the TaxJar sandbox or production authority according to the business's account, reports addresses the authority refuses without stopping the invoice, files the settled sale back under the invoice's number, and shows both sides' VAT registrations on invoices between European parties. The authority and ledger are reachable at TAX_JAR_URL, PROD_TAX_JAR_URL and INFLUX_URL, with an InfluxDB ledger in the sandbox.

Each task environment exposes only the services its workflow needs. The pool across tasks includes an AWS emulator, Docker, Kubernetes, GitHub, Linear MCP, PostgreSQL, MySQL, MongoDB, Gel, Redis, Go, Python, Node.js, Vitest, Slack, Intercom, Google Drive, Email and ClickUp. That is the practical difference from a repo-plus-test-suite benchmark: the agent has to work across code, infrastructure and business tools in the same rollout.

The instructions are not longer than comparable benchmarks; the work behind them is bigger. A median Real-SWE instruction runs 1,742 characters, against 2,056 for FrontierCode, 1,975 for DeepSWE, 1,584 for Terminal-Bench 3 and 992 for FrontierSWE v2. The median reference solution edits 11 files, compared with 6 for both FrontierCode and DeepSWE, using Cognition's published comparison for those two. Prompts are described as slightly underspecified, about par with DeepSWE and Terminal Bench, with any behaviour the verifier requires stated or reasonably discoverable.

Failure is not a function of giving up early. 71.4% of rollouts under 10 minutes failed (70 of 98), against 73.4% of rollouts of 10 minutes or longer (398 of 542). Those two buckets total 640 rollouts, which is ten tasks by eight models by eight runs. Specific Labs attributes the difficulty to triaging multiple systems and understanding requirements in codebases riddled with existing business logic and coding patterns.

In the published sample, 6 of 10 tasks resolve below 15%. The distribution is bimodal: Multi-region sweep resolves at 67.2% and API keys & environments at 65.6%, while Linearizable scan sits at 4.7%, Tax jurisdiction at 3.1%, and Analytics stream reducer at 0.0% across all 64 rollouts. No model solves every task.

Rankings also invert per task. On Customer identity migration (40.6% overall) Grok 4.6 passes 8/8 while GPT-6 Astra passes 1/8 and GPT-5.6 Sol 0/8. On S3 datastore measurement, Fable 5.1, GPT-6 Astra and Gemini 3.8 Flash all score 0/8 while GLM 5.3 gets 3/8 and Grok 4.6 2/8. Aggregate resolution rate is a weak predictor of which agent to point at a specific class of change.

Failures are grouped by observed submission behaviour using the same taxonomy across models, following DeepSWE: unverified assumption, missed requirement, integration error, regression, wrong file. Percentages are out of each model's failed runs, not all runs, and the published chart lists four values per model (three for Grok 4.6) against five labels, so the mapping below follows the legend order. Grok 4.6 attributes 67.2% of its failures to missed requirements and Kimi K3 53.8%; Gemini 3.8 Flash shows 49.1% integration error against 29.1% missed requirement. GPT-5.6 Sol leads on unverified assumptions at 43.3%, where Gemini 3.8 Flash is lowest at 10.9%.

Specific Labs' reading is that today's models are weaker at understanding company coding patterns and frequently miss requirements or do not verify their assumptions. For teams wiring agents into an enterprise repo, that puts the bottleneck at requirement capture and verification against existing conventions rather than at code generation. Harness-level checks that force an agent to enumerate and confirm requirements before submitting target the largest single failure bucket in this data.

The loudest objection on Hacker News is reproducibility. One commenter called it "pinky-promise benchmarking" and asked what the value is when nothing can be said about the code beyond that it was a large codebase from an unknown company. Others argued the trade is deliberate, that you give up transparency to make the benchmark harder to game, and that private benchmarks requiring trust in the tester already exist. Another said comparing models under the same tools is useful right now but poor for tracking progress over time, since it can get benchmaxxed or the underlying resources become obsolete.

Commenters also questioned the licensing mechanism, doubting that a company with a proprietary codebase worth studying would hand it to a third party, and speculating that the pool skews toward abandoned or small-to-medium codebases bought per line of code. The source says only that each codebase was licensed from a real-world company and screened for usage and production load; it does not name the companies or disclose licence terms. One commenter who runs a benchmark in another domain argued that many of the better private codebases may not be private anymore where Claude Code and Codex are concerned, and that contamination should be measured every time.

Gemini 3.8 Flash placing third drew direct disbelief, with one commenter saying it loops on the same file and makes hundreds of tool calls for a simple change. Two replies pointed at Antigravity, one saying the correct harness for Gemini 3.8 Flash is agy rather than Gemini CLI. Real-SWE ran it in Gemini CLI, so the 31.2% is specific to that pairing. One commenter also argued the leaderboard ignores latency and ergonomics, putting GPT-6 Astra at roughly 5x faster than Fable 5.1 in practice; the benchmark publishes no wall-clock or cost figures per model.

The public analysis covers a ten-task sample with eight rollouts per model, and the sample itself is available only on request. The prompt-length measurement is described as covering Real-SWE's eight repository-backed sample tasks, which does not match the ten tasks in the results table, and the release does not reconcile the two. The full task count, the number of licensed codebases, per-model cost and token usage, and the numeric confidence intervals are absent. There is also no comparable files-edited figure for Terminal-Bench 3 or FrontierSWE v2, so the 11-versus-6 gap holds only against FrontierCode and DeepSWE.

The thing to watch is whether the ordering holds as harnesses change underneath the models. Real-SWE scores pairs, so a Gemini result moving from Gemini CLI to another harness is a different measurement, not a correction. And because the tasks sit in private repositories, the usual external check, where someone reruns the eval and posts a diff, is unavailable by construction.