How well do agents use tests when you only name the technique?

The conditions covered essentially everything a non-expert might have heard of: ACL2, Alloy, Creusot, Kani, Lean 4, Spin, TLA+, Verus, SMT solvers with Z3, cvc5 and Yices all available, plus fuzzing, differential testing, metamorphic testing, mutation testing, property-based testing, QuickCheck, Proptest, Insta, rstest, the Rust built-in test framework, "Audit first", "Judgement" (agent picks the technique), and "Make no mistakes". All implementations were in Rust. The same pattern held on an IMAP RFC eval at 40 runs per condition and on a handful of other RFCs.

Mostly they perform the technique rather than apply it. Luu's summary is that agents either write the tests they would have written anyway but inside a framework belonging to a different technique, or use the technique superficially without doing the thing that produces value. With property-based testing, runs leaned on totally random inputs that hammered invalid and rejection cases, or picked a trivial property and threw low-value random cases at it. On xhigh, agents generally got their own tests to pass; the tests were just bad. One recurring example: submitting four identical bitstreams into a test of a feature that takes four bitstreams, which cannot catch a transposition bug.

At xhigh, fuzzing and property-based-testing conditions came out slightly ahead of formal methods on average; at medium the picture was much more mixed, and running at lower effort in a naive loop produced worse results across the board. Luu explicitly cautions against drawing strong conclusions from the per-condition ordering.

Verus uses an SMT solver to prove that code matches specifications. Agents given Verus and the Verus docs did not verify the implementation. They proved abstract properties about Zstd instead, and few of them: things like "given a valid cursor/index/distance, the resulting operation remains in bounds", which is fine but was not where the bugs were. Frequently the proofs were vacuous, restating the preconditions as postconditions.

The bug the runs kept shipping was failing to reverse the bitstream order between encode and decode, hidden behind palindromic tests. No proof addressed it. Aggregate xhigh Verus correctness came out fine but not better. Commenters on Hacker News read this as a data-volume problem: the quantity of Verus, TLA+, Creusot and Lean code in the world is tiny next to non-formal code, so an agent handed the tool without a spec has little to imitate beyond trivia.

Four skills were tested: the official Hegel skill, the ECC Rust test skill from a collection with 250k GitHub stars and 38k forks, the Trail of Bits property test skill, and a quick custom skill Luu wrote. The three recommended skills underperformed. The custom one did okay. The stated difference is design intent: the custom skill nudges agents away from their default behaviour toward more productive behaviours, while the others read like tutorials. The Hegel SKILL.md plus its linked Rust reference runs over 20k tokens. The ECC skill instructs agents to use TDD, and in the runs where agents actually followed that instruction, it fared poorly, consistent with the TDD condition itself.

The practical reading for anyone shipping an AGENTS.md or a skill file is that length and popularity are not the signal. In Luu's results, the skill that helped was the one written to push agents off a specific default, not the ones that explained a technique.

The strongest pushback in the Hacker News thread was about mutation testing. Several commenters read the writeup as describing manual mutation testing, where the agent writes code, writes passing tests, changes the code by hand, watches tests fail, then reverts, and asked why an automated mutation framework was not wired into the build to fail the build when mutants survive. As one commenter put it, an agent can ignore a TDD procedure but it cannot ignore a build failure. Another described gating at zero surviving mutants rather than a percentage threshold, and reported a case that morning where an agent-written test passed against both the fixed and the mutated code, looked fine, would have passed code review, and was caught only by the gate.

Others were less sure the gate holds: one asked whether automated mutation testing would help or whether agents would simply find new ways to game the metric. A separate line of criticism argued the experiment misses architecture, claiming 80% of effective testing lives in code structure, not the framework, and that forcing agents onto DI or hexagonal architecture plus a trivial coverage check produces good-enough code cheaply. The reply in-thread is that the same agent generated both the code and the tests, so the structure was chosen with the testing strategy already known. The sources do not test the architecture claim either way.

In a ten-way comparison of model and harness combinations on one Three.js hangar prompt, the columns that matter are "Opened in browser" and "Checked screenshots". SOL 5.6 Max under Codex did neither across 10m 48.765s and 1,097,441 total tokens. Astra 6.0 Max under Codex opened the browser but did not check screenshots, over 37m 29.705s and 1,335,495 tokens. Qwen 3.8 27B x-high under DSH did neither. GLM 5.3 Flash Max under Codex was blocked from opening a browser at all and logged 1 tool error. The runs that did both, Luna 5.6 Max on Codex, GLM 5.3 Flash Max on OMP and OpenCode, Qwen 3.8 27B on OMP, OpenCode and DSH/PTC, differ in harness, not only in model.

The failure mode without a gate is familiar. One developer describes receiving a 5,236-line vibe-coded payments backend PR from a business partner that simply did not work, because nobody ran the endpoints the agent had added to Bruno, which the developer says is two minutes of work. The accompanying documentation explained that errors return 400 and omitted where to get API keys.

If you are going to let agents run destructive verification steps such as installing fuzzers, running mutation frameworks, or spawning Docker, you want them somewhere disposable. Trail of Bits publishes coop, an Apache-2.0 Rust CLI that manages throwaway VMs for Claude Code and Codex, using Firecracker on Linux and Lima on macOS, tested on macOS arm64 and Linux x86_64, with Linux arm64 builds available but untested. The repository page shows 176 stars and 8 forks.

Luu's open question is why no lab appears to have built RL environments that teach agents to test well, given that agents have become good at bounded runtime optimization problems, exactly the kind of task you can generate cheap environments for. His guess is that knowledge of effective test technique is not widespread enough for anyone to have tried, or that packaging it is harder than it looks. Nothing in these results says whether an automated, build-failing mutation gate would fix the problem or just move it; that experiment has not been run. Nor do the sources cover the case several commenters actually want, where a human writes a non-trivial specification first, including liveness, and the agent produces an implementation by guided refinement.