the short version
- A small transformer trained from scratch at test time scores 44% on the ARC-AGI-1 public eval for 67 cents and 7% on ARC-2.
- Representation carried the result: removing 3D RoPE or the per-task embedding each drops the score to roughly 24%.
- Test loss got worse when the author stopped training on input tokens, yet the score rose from 40% to 44%, which is a direct warning against tuning on validation loss alone.
- The union of solved tasks across multiple runs reaches 55%, so candidate selection, not capability, is the binding constraint.
Published
Key facts
- 44% on the ARC-AGI-1 public eval for 67 cents, up from a previous 40% result; the same run scores 7% on ARC-2
- The transformer trains from scratch at test time in 1.5 hours on a single RTX 5090, with no frontier LLM in the loop
- Dropping input tokens from the loss, making the run supervised, moved the score from 40% to 44%
- Ablations: 1D RoPE instead of 3D drops to ~24%, removing per-task embeddings drops to ~24%, CompressARC-style per-task training drops to ~18%
- The union of solved tasks across multiple runs reaches 55% versus 44% for a single run, an 11-point selection gap
Mithil Vakde reports 44% on the ARC-AGI-1 public eval for 67 cents, using a small transformer trained from scratch in 1.5 hours on a 5090. The same run scores 7% on ARC-2. There is no frontier LLM in the loop: the model is trained at test time on the train set and eval set puzzles with test labels hidden, and the code is open source.
This is the third post in a series, and an upgrade on a previous result that scored 40%. The comparison chart in the post is restricted to models that do similar test time training, not to LLM-based ARC solutions. The author's stated motivation is sample efficiency, and the reason for the cost obsession is iteration speed rather than a production budget.
What the 67-cent recipe actually is
- Each input-output pair is converted to a sequence of tokens, and those sequences are autoregressively trained on by a small transformer from scratch at test time.
- Each puzzle gets a separate learnt additive embedding, which is what enables cross-task learning inside a single run.
- Because each sequence holds two 2D grids, positions are learnt with 3D RoPE embeddings.
- The sequences are augmented with color and dihedral permutations; at inference the test inputs are augmented and the inverse augmentation is applied to the outputs produced.
- The two most common outputs across those inverted candidates are submitted, an approach the post calls AAIVR.
That last step is the whole candidate-generation and voting story. There is no separate verifier model, no scoring head and no reranker. Diversity comes from structure-preserving transformations of the input rather than from temperature, and the aggregation is a plain frequency count over decoded grids after the transformation is undone.
Which changes bought score and which bought cost
The post separates the two, which is useful because the levers are different. Score came from modelling choices; cost came from kernels, optimizers and doing fewer augmentations rather than more.
- Score: a modern architecture (SwiGlu instead of GELU, RMSnorm not layernorm), more data diversity with better shuffling, and scaling from 4 layers to 8.
- Cost: way fewer augmentations, which the author frames as being more sample efficient; AdamW replaced by NorMuon; flash attention with varlen training plus flex attention kernels for inference.
- Data: non-overlapping tasks from ARC-2 were added, with the 773 ARC-1 puzzles that ARC-2 repeats filtered out to avoid a leak. Removing that extra data still scores about 40% but needs roughly double the compute.
- Optimizer detail worth stealing: vanilla Muon trained much faster than AdamW but the loss and scores would loiter at the end instead of converging. Cranking momentum or learning rate down helped manually; switching to NorMuon made the problem disappear.
One change is doing more work than its size suggests. The author stopped training on input tokens, so the loss now only includes output tokens, which makes the approach supervised. That moved the score from 40% to 44%, and the post is explicit that he does not understand why, offering finite model capacity only as a guess.
The ablations say representation is the whole game
- Switching 3D RoPE to 1D drops the score to about 24%.
- Removing the per-task embeddings drops the score to about 24%; both of those ablations saturate around 25%.
- Training on inputs performs slightly worse, at about 39%.
- Restricting the training set to ARC-1 plus ConceptARC lands at about 40%, roughly the same.
- Running the model CompressARC style, training from scratch on each task separately and unsupervised, drops to about 18%; CompressARC but supervised gets about 15%.
The author's own summary is that the biggest contribution to performance seems to be good representations, meaning 3D RoPE plus the per-task embedding. Note the gap between the per-task-embedding run and the per-task-training runs: sharing one model across all puzzles with a learnt task vector beats training a separate model per puzzle by more than 20 points. Costs across ablations are not comparable, since all but the first require a lot more compute.
What transfers to ordinary agent workloads
The transferable part is the decoding loop, not the ARC-specific model. Generate candidates by applying invertible transformations to the input, run each through the model, invert the transformation on the output, then take the most frequent answers. That is self-consistency with structured perturbations instead of sampling noise, and it applies anywhere an agent's input has symmetries the answer should be invariant to. The cost is linear in the number of augmentations, and the post's finding that fewer augmentations both cut cost and improved sample efficiency argues against cranking the candidate count up by default.
The second transferable finding is a warning about how you tune. With the supervised change, test loss got worse while the score improved, and the runs became more stable with less variance in scores. The author calls this out directly as a failure mode for people optimizing lowest val loss on a small dataset. Any team tuning an agent on a proxy loss or an LLM-judge score rather than the end task should treat this as evidence that the proxy can move the wrong way.
The third is a sizing signal for verification. The author took the union of all solved tasks from multiple runs and got 55%, against 44% for a single run, with a further set of tasks described as almost solved. Eleven points sit inside candidates the system already produced but did not select. In agent terms, that is the gap a better verifier or selection stage is competing for, and it is measurable before you build one.
What the post does not tell you
There is no latency number, no breakdown of how the 67 cents is computed, and no per-task wall-clock figure; the only timing given is 1.5 hours on a 5090 for the whole run. The 44% is on the ARC-AGI-1 public eval, and the ARC-2 score is 7%, so nothing here says the recipe generalizes past grid puzzles. The author says he is still evaluating whether unsupervised training is better in some scenarios, and invites others to try for 65% on the open-source code, with the request not to increase training data.
Questions this raises
how does test-time training get 44% on ARC-AGI-1?
Each input-output pair is turned into a token sequence and a small transformer is trained autoregressively from scratch at test time on the train and eval puzzles with test labels hidden. Each puzzle gets its own learnt additive embedding and positions use 3D RoPE. At inference the test inputs are augmented with color and dihedral permutations, the inverse augmentation is applied to each output, and the two most common grids are submitted.
what is AAIVR in the ARC-AGI test-time training post?
AAIVR is the author's name for the decoding loop: augment the test input, run it through the model, invert the augmentation on the output, then vote by frequency over the decoded grids. There is no verifier model, scoring head or reranker, and diversity comes from structure-preserving transformations rather than sampling temperature.
why did test loss get worse while the ARC score improved?
When the author stopped training on input tokens, test loss went up but the score rose from 40% to 44% and the runs became more stable with less variance. The post flags this as a failure mode for anyone selecting checkpoints by lowest validation loss on a small dataset, and the author says he does not fully understand why it works, guessing at finite model capacity.
These daily notes are drafted by a model I run and operate myself - the same kind of pipeline this site is about - from sources published in the previous 24 hours, and every one lists what it read. The longer essays, the talks and the preprint are mine, written by hand.