Auto-Research Arena benchmark: a simple sequential search beats sophisticated harnesses

The baseline finding matters most for anyone building research agents. A plain sequential loop beat more elaborate harnesses on a benchmark that refuses to rank results failing a quality and resource gate. On that benchmark, at least, the elaborate parts of those systems are not yet producing better results. The abstract makes no claim about the reliability of any system and reports no cost figures.

The authors describe Arena as a multi-target benchmark with two jobs. It evaluates autonomous research systems, and it is also a way to discover ideas in language modeling that can be reused in new searches.

The core design decision is to separate admissibility from improvement. Arena supports diverse research objectives within a shared experimental environment. A result is admissible only if it meets the model-quality and resource constraints. The selected target then measures progress only among admissible results.

This structure targets a familiar failure in automated optimization. An agent asked to make inference faster can do so by giving up model quality, and an agent asked to cut data use can do so by training a worse model. Under the Arena split, such results fail the gate and are never ranked on their target. The abstract does not give the quality thresholds or the resource budgets, and it does not say how strictly either is enforced. A reader cannot yet tell how hard the gate is to pass.

Arena is instantiated in language model pretraining with nine targets. The abstract describes them as spanning model, data, training, and inference efficiency. All nine sit in the same shared experimental environment.

The abstract does not list the nine targets individually or say how they divide among these areas. It also does not give the model scale, the dataset, or the compute budget used in the pretraining environment. Those details determine whether a discovery is likely to transfer to a larger production training run. Until the full paper is read, treat the scale question as open.

The evaluation covers seven research harnesses using two foundation models. The abstract does not say whether every harness was paired with both models. According to the authors, Arena clearly differentiates existing harnesses across research objectives. Several sophisticated methods were outperformed by what the authors call "our simple sequential search baseline".

The abstract does not name the seven harnesses or the two foundation models. It does not say how many of the sophisticated methods lost to the baseline, by what margin, or on which of the nine targets. It does not report run cost or wall-clock time for any harness. Without those figures the result is a qualitative ranking, and it cannot yet support a decision such as dropping a specific framework.

The paper's own reading is about headroom, not trust. Still, the design bears directly on trust. Because the admissibility gate comes first, any reported improvement has already cleared a quality and resource floor. That is a stronger claim than movement on the target alone.

The baseline result points the other way. When a simple sequential search beats more sophisticated harnesses, the extra machinery in those harnesses is not reliably finding better results. Teams evaluating an auto-research tool can take several practical points from this.

Beyond evaluation, the authors report that Arena reveals a substantially broader frontier of model and system ideas, which they describe as aligned with active research directions. They then reuse these discoveries in a new quality-driven search. The available abstract cuts off partway through the sentence that reports the outcome, ending at "yields lower traini", so this note does not state what the reuse improved or by how much. The abstract also does not list the discovered ideas, so it is not yet possible to judge whether they are novel or rediscoveries of known techniques.

For engineers building agent systems that run experiments, the transferable part is the evaluation structure rather than the rankings. Defining what counts as an admissible result separately from what counts as progress is a pattern that applies outside pretraining. It also makes reported gains easier to defend in review.