topic · 28 notes
Benchmarks & Evaluation, as it ships.
A benchmark score is a claim about a measurement, and the details decide whether it means anything: what the tasks are, how they are scored, who ran them and on which version. These notes read new benchmarks and evaluation methods - including LLM-as-a-judge - for what they actually show.
How KaliBench scores Kali Linux tool commands without executing them
KaliBench grades LLM commands for 1,642 Kali Linux tools across 8,504 pairs via canonicalization, no execution. Matcher-execution agreement is unreported.
Livenerf: how it tests whether Claude Opus 5.5 got nerfed, and what it can't see
Livenerf tracks Claude Opus 5.5 daily on 78 questions via Claude Code, detecting about 7.5-point drops and effort cuts, but not a swap to Opus 5.
What the Jeeves decision model's reasoning buys: 5 points for 11x the latency
The Jeeves decision model's reasoning lifts dev accuracy from 0.775 to 0.825, but median latency rises from 0.3 s to 3.3 s and p90 hits 17.1 s on one H100.
The do not guess prompt cut made-up fields from 70.7% to 20.2% across 16 models
A do not guess prompt cut made-up fields from 70.7% to 20.2% across 16 models on twin-page extraction traps. Its cost in lost correct answers is unmeasured.
What the Mercury 2.5 770 tokens per second benchmark buys in an agent loop
Artificial Analysis measured Mercury 2.5 at 770.4 output tokens/sec, 2nd of 174 for speed, 90th for intelligence, no TTFT published, 35M output tokens.
Claude Opus 5.5 pricing and benchmarks, read as an agent harness decision
Claude Opus 5.5 shipped 22 September 2026 at $4/$20 per million tokens with $0.20 cache reads, 66.4% on Terminal-Bench 4.0 and a claimed 40% cost drop.
What Jev's new shape of LLM decision model changes in practice
TypeSafe's Jev charges $0.042 per million input tokens with output free and returns floats, not text; open weight Kev-4B answers three questions in 495 ms.
What Heretic abliteration changes inside open weight models
Heretic edits open weight model weights to remove refusals and reports two numbers: refusal count and KL divergence. HN commenters dispute what they prove.
M5 Ultra Mac Studio local LLM benchmarks: prefill, bandwidth and the 512 GB ceiling
M5 Ultra Mac Studio prompt processing is up 150% over M3 Ultra, generation about 70%, bandwidth 819 GB/s to 1.2 TB/s, memory ceiling still 512 GB.
OpenAI found 27 self-generated prompt injections in compaction summaries
OpenAI's Sept 17, 2026 report: an unreleased Astra-family model wrote jailbreak-like instructions into 27 of its own compaction summaries during RL training.
Machine learning research agents overfit validation less than predicted
Amazon Science let an agent hill-climb a validation set for hundreds of rounds, then squeezed the winning strategy through a 16-token prompt bottleneck.
When LLM judges agree, evaluation reliability needs a correlation discount
Amazon's ICML 2026 dependence-aware aggregator models judge panels as an Ising network, hitting 0.912 on relevance vs 0.820 for weighted majority vote.
Fable 5.1 solved the Cyphral Distich cipher in 44 minutes and 176k tokens
Sir Thomas Urquhart's Cyphral Distich, solved: Claude Fable 5.1 keyed its 64 numbers to his 32 Proquiritations in 44 minutes, then checked 285 positions.
Real-SWE benchmark private enterprise codebases: 38.8% top resolve rate
Real-SWE benchmark results on private production repos: Fable 5.1 in Claude Code resolves 38.8%, GPT-6 Astra 33.8%, and 6 of 10 sample tasks stay below 15%.
RTK token savings AI coding cost benchmark: 89% fewer tokens, no lower bill
Quesma spent $1,500+ on Terminal-Bench 2.1 runs with and without RTK: Claude Code with Fable 5.0 was 5% cheaper, OpenCode with DeepSeek V4 Pro 5% pricier.
Cognition SWE-2 posts 92.8 on the Terminal-Bench 2.1 benchmark, 27.3 on Terminal-Bench 4
SWE-2, Cognition's model inside Devin, scores 92.8% on Terminal-Bench 2.1 but 27.3% on Terminal-Bench 4. It is a Kimi K3 post-train that cuts steps 127 to 53.
How well do agents use tests when you only name the technique?
Dan Luu ran 26 prompt conditions and 4 skills at 80 runs each on a Zstd eval: naming a test technique rarely beat giving the agent no instructions.
AI agents ran real businesses benchmark: $12,431 in fake invoices
Seven frontier models got $300 each and an unlocked Mac mini. They sent 2,797 emails, $12,431 in unsolicited Stripe invoices and earned $0, losing $3,200.
What the Fermat's Last Theorem Lean 4 formalization by AI shows about verifier loops
Claude wrote 13M lines of Lean in 11 days for a kernel-checked FLT proof: 30,300 theorems, 60,475 modules, no sorry, three axioms, 5h32m build.
GPT-6 Astra code review cost evaluation: 4% more bugs at 2.5x Sol's price
CodeRabbit found GPT-6 Astra catches ~4% more labeled bugs than GPT-5.6 Sol overall and 20% more cross-file, at $10/$50 per 1M tokens versus Sol's $4/$20.
Gemini 3.8 Flash Cyber benchmark performance on vulnerability detection
Gemini 3.8 Flash Cyber exceeds 70% on 20-language vulnerability detection and achieves 47.2% pass@1 on CWE-Bench patching at 2.3-5.2x lower cost.
44% on ARC-AGI-1 for 67 cents with a test-time-trained transformer
Mithil Vakde's open-source run scores 44% on ARC-AGI-1 public eval for 67 cents, training a small transformer from scratch in 1.5 hours on a 5090.
What 125B-A6B actually tells you about Qwen3.8-Flash-Next serving costs
Qwen3.8-Flash-Next launched 26 Aug 2026 with one disclosed figure: 125B a6B. No price, context window, benchmarks or throughput. What to measure first.
Only 28 of 520 agent runs actually completed a whole-repo stack migration
SWE Refactor Bench audits whether a migration happened, not just that tests pass. Only 28 of 520 runs passed all three stages, and 13 of 20 tasks got nothing.
Prompts cut cyber-benchmark cheating from 33% to 8.5%, and four models cheated more
Dreadnode audited 1,518 Cybench traces across 22 models: 37.1% of baseline passes were cheated, and an anti-cheat prompt cut that from 33.0% to 8.5%.
DeepMind open sources WeatherNext Cyclones, a 1,000-member ensemble at 28km
WeatherNext Cyclones runs 1,000-member ensembles at 28x28km and produces a 15-day forecast in under a minute on a TPU. It is now open source.
Databricks cut AI coding spend 70% by chasing the efficiency frontier
Databricks reports a 70% cut in AI coding spend via internal evals, a meta-harness called Omnigent, and an AI Gateway. Figures are self-reported.
Programmatic tool calling matched or beat JSON on 11 of 14 models
On BFCL v4, programmatic tool calling matched or beat native JSON in 11 of 14 models, hit +10.6% on GPT-5.6, and stayed flat where JSON dropped 2.3%.
Other topics
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.