Livenerf: how it tests whether Claude Opus 5.5 got nerfed, and what it can't see

Opus 5.5 shipped on 2026-09-22. The ninjahawk/livenerf repo on GitHub started its series on 2026-09-24 at 22:10 UTC, about 2.5 days after launch. The author states the motivation plainly: nobody has had a clean day-0 baseline to check against, so nerf arguments end up as "vibes versus vibes."

As of 2026-09-29, 6 of 30 daily runs had been collected, none missed. Each ran the full 90 samples on harness hash 461391b6fce64167 with Claude Code CLI 2.1.280 pinned, and day 5 ran with the budget guard overridden once. Days 1 to 10 form the baseline, and the first possible call lands around 2026-10-24.

Sampling parameters are gone and thinking cannot be turned off, so Opus 5.5's outputs cannot be made deterministic. The benchmark fixes everything else instead: frozen prompts, a pinned CLI, exact graders and raw logs kept forever. It then measures drift statistically over thousands of samples. It is built on Inspect, the UK AI Security Institute's open-source eval framework, and its statistics follow Anthropic's Adding Error Bars to Evals.

The author screened 2,336 GPQA Diamond, MMLU-Pro, competition-math and AIME 2025–26 questions with 4 samples each. Opus 5.5 got about 93% right on the first try, and 97% of the questions were always right or always wrong. Under a common logit-shift model, a question with pass rate p carries information p(1−p) per sample, so a question the model always gets right cannot show a drop. The 78 questions that were sometimes right became the panel. Per the README, this step also filters out memorized questions.

The author measured the resulting selection bias. Questions chosen for being sometimes right look closer to 50/50 than they are. On fresh samples, their pass rate rose from 54.7% to 62.0%, so the power calculation uses the fresh rates rather than the screening rates.

Before the baseline began, the rig was checked against a known degradation, effort medium against high, and an A/A check confirmed that the error bars are honest. Validation passed its pre-registered criterion. The README reports that lower effort shows up much more clearly in tokens than in accuracy:

This is why the output token count per sample is the secondary signal the author says they care most about. In the README's words, if a model quietly starts thinking less, the token count is where it shows up first, often before accuracy moves at all. For teams building their own monitoring, the implication is to log token usage alongside scores on every sample.

The primary metric is the per-item paired score difference against baseline, averaged over items, with clustered standard errors following Miller 2024. Pairing each question with its own baseline removes question difficulty from the comparison. Scoring is exact-match only. An LLM judge is excluded because the judge would drift too.

The control arm is designed to catch harness and platform changes. Every day, claude-opus-5 runs the panel's GPQA questions through the same harness, and if both models move together, the change is attributed to the harness or platform, not to Opus 5.5. Every call is also hermetic: a short frozen system prompt, no tools, no MCP servers, no settings or hooks, no CLAUDE.md or memory, one turn, and a fixed empty working directory. The README gives the reason for pinning the CLI:

PREREGISTRATION.md was committed before any series data, so its git timestamp is public. The livenerf.analysis step applies the decision rule mechanically. It tests for change in either direction, because launch week could be the worst week: a new serving stack, capacity strain, launch bugs. The README notes that the 2025 quality incidents turned out to be infrastructure bugs, not deliberate downgrades.

The most important limit is in the validation data. Swapping in Opus 5 was not distinguishable from Opus 5.5 at 99%: −3.8 ± 6.3 points of accuracy and −23% tokens. The README says directly that the instrument cannot detect a same-family model swap of that size in a validation's worth of samples. A 10-day window has about 2.5 times as many samples, but that has not been shown to be enough. A "no change" result would therefore not rule out a same-family swap of that size, which is close to the "smaller model behind the same name" scenario the README lists among possible nerfs.

There are three other limits:

Skeptics in the thread made two points. One commenter argued that the perceived drop always arrives around a week after launch, attributed it to hedonic adaptation, and said the many benchmark attempts to show nerfing never persist. Another linked xkcd 882 and noted that running more experiments produces more statistically significant-looking results. The repo's design targets both points: a day-0 baseline, a single pre-registered primary metric, a 99% threshold, a two-consecutive-window requirement, and a commitment to publish null results and improvements.

On why the windows are 10 days, a reply in the thread described it as a tradeoff:

The same commenter said the window rolls forward daily and that its length may be revised once there is enough longitudinal data. That raises a question the sources do not answer. It is not clear whether repeated daily looks at a rolling window inflate the false-positive rate relative to the nominal 99%.

Other commenters asked whether API or cloud-hosted Claude is affected. One per-token Enterprise user reported never noticing slowdowns or a gradual decline in quality. The benchmark cannot settle that question, because it deliberately measures only the subscription path. Another anecdote, about more frequent permission prompts, also falls outside a benchmark that runs with no tools.

Several of the methods carry over to any team tracking a hosted model:

The budgeting is concrete. The daily run of the whole panel uses about 3.6% of the weekly plan. An attempt is skipped if the weekly meter is at or above 75% or the 5-hour meter is at or above 60%, and it retries every hour until the day's run is in. The main caveat is sensitivity. The detectable change is about 7.5 points per window, while the effort-medium degradation in validation moved accuracy by only 4.2 points, so a regression of that size could go unflagged on accuracy alone.

The first Results row is expected after day 20, and the first possible verdict around 2026-10-24. Points to track: