topic · 8 notes
Training & Fine-Tuning, as it ships.
Notes on how models are trained and adapted after pre-training: supervised fine-tuning, reinforcement learning, distillation and test-time training - what each method needs in data and compute, and what the published results show.
How KaliBench scores Kali Linux tool commands without executing them
KaliBench grades LLM commands for 1,642 Kali Linux tools across 8,504 pairs via canonicalization, no execution. Matcher-execution agreement is unreported.
Clef decision models ship Apache 2.0 weights, a 38.8 ms flash tier and RL fine-tuning
Clef decision models from Cloudflare ship Apache 2.0 weights on Qwen backbones, a 38.8 ms Clef-flash tier vs Jev's 524.1 ms, and FDE-run RL fine-tuning.
What the Jeeves decision model's reasoning buys: 5 points for 11x the latency
The Jeeves decision model's reasoning lifts dev accuracy from 0.775 to 0.825, but median latency rises from 0.3 s to 3.3 s and p90 hits 17.1 s on one H100.
What Jev's new shape of LLM decision model changes in practice
TypeSafe's Jev charges $0.042 per million input tokens with output free and returns floats, not text; open weight Kev-4B answers three questions in 495 ms.
How a 4B model trained with RL produced query plans faster than Postgres
A 4B Qwen model post-trained with SFT and a custom GRPO variant hit a 1.81x geometric mean speedup and 44.7% lower summed latency on 113 join-heavy queries.
Machine learning research agents overfit validation less than predicted
Amazon Science let an agent hill-climb a validation set for hundreds of rounds, then squeezed the winning strategy through a 16-token prompt bottleneck.
Cognition SWE-2 posts 92.8 on the Terminal-Bench 2.1 benchmark, 27.3 on Terminal-Bench 4
SWE-2, Cognition's model inside Devin, scores 92.8% on Terminal-Bench 2.1 but 27.3% on Terminal-Bench 4. It is a Kimi K3 post-train that cuts steps 127 to 53.
44% on ARC-AGI-1 for 67 cents with a test-time-trained transformer
Mithil Vakde's open-source run scores 44% on ARC-AGI-1 public eval for 67 cents, training a small transformer from scratch in 1.5 hours on a 5090.
Other topics
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.