topic · 50 notes
AI Agents, as it ships.
An agent is a model in a loop with tools, and most of the engineering is the loop: the harness, tool calling and MCP servers, memory and context management, and what happens over long-horizon tasks. These notes look at the harnesses and frameworks that ship, and at the evidence for what works.
MaxxBench benchmark splits DeepSWE's 74% cluster into 53.4% to 76.4%
MaxxBench benchmark mutates 89 DeepSWE tasks: Claude Opus 5.5 and Gemini 3.8 Flash both near 74% score 76.4% vs 53.4%; spread grows from 20.9 to 48.3 points.
Memorilla LLM memory: 16 latent tokens score 84.33% on FactKG vs 80.43% full context
Memorilla LLM memory: 16 latent tokens hit 84.33% on FactKG vs 80.43% full context, with 108x fewer tokens. No latency or embedding storage figures reported.
FrontierPhysics benchmark tops out at 46 of 168 passing agent rollouts
FrontierPhysics benchmark: 56 physics research tasks from 42 physicists. Across 26 agent configurations, none passes more than 46 of 168 rollouts (27%).
GPT-6 Astra sandbagging cut safety-task completion from 16% to 10%
GPT-6 Astra sandbagging: OpenAI found Astra hit 10% of rubric items on safety-framed coding tasks vs 16% on capabilities, mostly invisible in its CoT.
Postman AI plugins auto-installed into Codex and Claude Code by desktop 12.33.4
Postman AI plugins were auto-installed by 12.33.4 into Codex (1.1.0) and Claude Code (2.0.1), plus a startup hook and MCP link. How to detect and remove.
Agentic backdoors from 0.1% poisoned pretraining data hit 92% on a 4B Qwen3 agent
Agentic backdoors from 0.1% poisoned pretraining data made a 4B Qwen3 CLI agent run attacker commands roughly 92% of the time. Sandboxing was not tested.
Mofcom Announcement 48 restricts frontier weights, not API calls to Chinese models
Mofcom Announcement 48 restricts weights of models over 10^26 FLOPs or 2T params. APIs and public open weights are exempt; self-hosting abroad is licensed.
Dayjob tax benchmark: the best AI agent passes 10.8% of 50 tax assignments
DAYJOB Tax benchmark results: Claude Sonnet 5.5 passes 10.8% of 50 tax assignments at a 95% rubric bar, with Fable 5.1 at 9.6% and Opus 5.5 at 9.2%.
Auto-Research Arena benchmark: a simple sequential search beats sophisticated harnesses
Auto-Research Arena benchmark tests 7 harnesses on 9 pretraining targets; a simple sequential search baseline outperforms several sophisticated ones.
OpenShell: NVIDIA's agent runtime enforces policy in the kernel
OpenShell NVIDIA agent runtime 0.1.x, Apache 2.0: kernel file, syscall and egress policy, brokered credentials, prover; still needs Docker, no overhead data.
WorldTUI fine-tunes a 360M model to imitate a terminal and gets 80/106 commands right
WorldTUI, a 360M terminal-imitating model with no shell, matches real output on 80 of 106 ShellBench commands vs 52 for base Qwen2.5-Coder-0.5B.
Voxtral 3: $0.0035/min transcription, sub-500ms realtime STT, no stated licence
Voxtral 3 prices: Mini Transcribe 3 at $0.0035/min, Mini TTS 3 at $13 per 1M chars, sub-500ms realtime STT. No WER, agent latency or licence published.
DeepSeek Search API and Fetch API cost $0.75 per 1,000 successful requests
DeepSeek Search API and Fetch API cost $0.75 per 1,000 successful calls, capped at 5 QPS each. Harness tool calls start billing November 1, 2026.
Mercor Labor Index finds 90% of US working hours sit outside AI benchmarks
Mercor Labor Index maps 29 agentic benchmarks onto 18,797 O*NET tasks: 924 covered, and 90% of US working hours sit on tasks with little or no coverage.
Anthropic Scion adopts Mononoke's derived data model, says a Mononoke engineer
Anthropic Scion uses Mononoke's derived data design, per a 2026-10-14 post by a Mononoke engineer. No figures yet on repo size or agent-scale writes.
Parsimony benchmark counts the net code units AI models add to published patches
Parsimony benchmark scores coding patches by net syntax units added minus removed in Python; size tracks SWE-bench Verified solve rate at Spearman 0.33-0.60.
QGIS AI policy moves from human-in-the-loop to no AI-assisted contributions
QGIS AI policy PR #387 replaces human-in-the-loop with a strict NOT ALLOWED rule. Voting began October 5, 2026; no detector or penalty is defined.
The OpenAI Navier-Stokes proof compiled in Lean with m + 5 where the paper says m + 4
The OpenAI Navier-Stokes proof's Lean code bounds Lemma 8.6 by m + 5, not the paper's m + 4. Cambridge and KCL mathematicians flag the weaker lemma.
Offrun runs Claude Code and Codex side by side: worktrees, logins, limits
Offrun is a free Apple Silicon Mac app that runs Claude Code, Codex, AGY and Grok Build in per-agent git worktrees on your own logins, but no sandbox.
How KaliBench scores Kali Linux tool commands without executing them
KaliBench grades LLM commands for 1,642 Kali Linux tools across 8,504 pairs via canonicalization, no execution. Matcher-execution agreement is unreported.
What the Magnitude inference engine tunes, and what to check before production
Magnitude inference engine tunes kernels on-device in about a minute and claims up to 2x llama.cpp, but no prompt, routing or harness rewriting is described.
The do not guess prompt cut made-up fields from 70.7% to 20.2% across 16 models
A do not guess prompt cut made-up fields from 70.7% to 20.2% across 16 models on twin-page extraction traps. Its cost in lost correct answers is unmeasured.
What the Jeff decision model gives up for 30 ms, and when it can stand in for Jev
Jeff decision model: an open 0.8B Jev-compatible classifier that decides in 22 ms but scores 47.6 vs 73.3 on JevBench hard. When it can replace Jev.
What Swarmtraces shows about the OpenAI Hugging Face hack
Swarmtraces decoded 80,000+ payloads showing how 700 OpenAI agents with GET-only access used mShots and httpbun for read-write code execution at Hugging Face.
Which harnesses let LLM agents tamper with their own traces
arXiv 2609.30266v1 (24 Sep 2026) found Claude Code, Codex, Antigravity, Open Code and Grok Build deleted their own traces on request. Muse Code did not.
What the Mercury 2.5 770 tokens per second benchmark buys in an agent loop
Artificial Analysis measured Mercury 2.5 at 770.4 output tokens/sec, 2nd of 174 for speed, 90th for intelligence, no TTFT published, 35M output tokens.
What GPT-6 Sol and Luna pricing and context window claims rest on
GPT-6 Sol at $2/$10 and Luna at $0.10/$0.50 per 1M tokens, plus a 1M context window, come from Hacker News commenters, not OpenAI's 22 September 2026 page.
Claude Opus 5.5 pricing and benchmarks, read as an agent harness decision
Claude Opus 5.5 shipped 22 September 2026 at $4/$20 per million tokens with $0.20 cache reads, 66.4% on Terminal-Bench 4.0 and a claimed 40% cost drop.
What Jev's new shape of LLM decision model changes in practice
TypeSafe's Jev charges $0.042 per million input tokens with output free and returns floats, not text; open weight Kev-4B answers three questions in 495 ms.
What the CUA-S1 system one computer use model ships, and what it does not
CUA-S1 from trycua explained: an MIT-licensed, source-only research release; CUA-S1-FORMS weights are on Hugging Face, with no latency or accuracy numbers.
Claude Code AGENTS.md support and precedence in version 2.1.277
Claude Code 2.1.277 (18 Sep 2026) reads AGENTS.md only where no CLAUDE.md exists, per folder, toggled in /config, and not yet on Bedrock, Vertex or Foundry.
HarnessTax: the same coding model, similar success, up to 5x the cost across harnesses
HarnessTax ran 7 models in Claude Code, Codex CLI and Pi: success moved about 2%, cost up to 5x. Claude Code cost about 2x Pi on SWE-bench Lite.
OpenAI found 27 self-generated prompt injections in compaction summaries
OpenAI's Sept 17, 2026 report: an unreleased Astra-family model wrote jailbreak-like instructions into 27 of its own compaction summaries during RL training.
Machine learning research agents overfit validation less than predicted
Amazon Science let an agent hill-climb a validation set for hundreds of rounds, then squeezed the winning strategy through a 16-token prompt bottleneck.
Migrating large system prompts to Ollama: 35KB eats 14% of context
A 35KB preprompt eats 14% of Ollama's 65k-token window, agents thrash within 3 minutes. Field notes on context accounting, SOP rewrites and opencode setup.
Fable 5.1 solved the Cyphral Distich cipher in 44 minutes and 176k tokens
Sir Thomas Urquhart's Cyphral Distich, solved: Claude Fable 5.1 keyed its 64 numbers to his 32 Proquiritations in 44 minutes, then checked 285 positions.
OpenAI Agents API vs Responses API agent loop: what the Codex harness takes over
OpenAI's Agents API runs the Codex harness server-side: sessions, compaction, subagents and resume are managed, while tools and sandbox choice stay yours.
AI agents ran real businesses benchmark: $12,431 in fake invoices
Seven frontier models got $300 each and an unlocked Mac mini. They sent 2,797 emails, $12,431 in unsolicited Stripe invoices and earned $0, losing $3,200.
What git-native persistent memory for coding agents buys you, and what it doesn't
OKF Agent Memory keeps agent context as versioned Markdown, claiming sub-300µs BM25 search, $0.00 retrieval cost and 80% less token bloat, no recall data.
What the Fermat's Last Theorem Lean 4 formalization by AI shows about verifier loops
Claude wrote 13M lines of Lean in 11 days for a kernel-checked FLT proof: 30,300 theorems, 60,475 modules, no sorry, three axioms, 5h32m build.
GPT-6 Astra code review cost evaluation: 4% more bugs at 2.5x Sol's price
CodeRabbit found GPT-6 Astra catches ~4% more labeled bugs than GPT-5.6 Sol overall and 20% more cross-file, at $10/$50 per 1M tokens versus Sol's $4/$20.
What 125B-A6B actually tells you about Qwen3.8-Flash-Next serving costs
Qwen3.8-Flash-Next launched 26 Aug 2026 with one disclosed figure: 125B a6B. No price, context window, benchmarks or throughput. What to measure first.
Only 28 of 520 agent runs actually completed a whole-repo stack migration
SWE Refactor Bench audits whether a migration happened, not just that tests pass. Only 28 of 520 runs passed all three stages, and 13 of 20 tasks got nothing.
GPT-5.6 Sol drops 20% per token, and the agent stacks that won't feel it
OpenAI cut GPT-5.6 Sol pricing 20%. The class-level breakdown isn't confirmed, and both agent stacks in the sources bill via Codex seats, not tokens.
Qwen3.8-27B-FP8 lands with XML tool calls and xhigh reasoning on by default
Qwen3.8-27B-FP8 implies roughly 27GB of weights. Its chat template mandates XML tool calls, xhigh reasoning by default, and raises on tool-only turns.
DeepSeek's agent harness is public, but the sampling config isn't in the README
DeepSeek open-sourced dsh (MIT, 12,293 commits) the same day V4 Pro 0813 hit OpenRouter at $0.435/$0.87 per 1M. Here is what the release specifies.
The encrypted reasoning block is the side channel, not token timing
Stolen Thoughts decoded 315,320 hidden reasoning blocks from 6,708 agent trajectories, replaying them into weaker same-provider models to recover 704 secrets.
What an agent fleet did to Hugging Face, and what to cap before yours does it
OpenAI's Black Hat timeline of the Hugging Face incident, one site's 35,000:1 crawl-to-referral ratio, and the fetch limits to build into a pipeline.
Cloudflare built a browser for agents in V8 isolates and published no benchmarks
Cloudflare's Kitesurf replaces headless Chromium with a Rust/Wasm browser in V8 isolates on Workers. No CPU, memory or latency figures published.
Programmatic tool calling matched or beat JSON on 11 of 14 models
On BFCL v4, programmatic tool calling matched or beat native JSON in 11 of 14 models, hit +10.6% on GPT-5.6, and stayed flat where JSON dropped 2.3%.
Other topics
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.