# FrontierPhysics benchmark tops out at 46 of 168 passing agent rollouts

> A new set of 56 end-to-end physics research tasks grades agents on correctness and publishability. No configuration out of 26 passes more than 46 of its 168 rollouts.

- Published: 2026-11-05
- By: Samir Sengupta (https://www.samcodeman.com/about)
- Canonical: https://www.samcodeman.com/writing/frontierphysics-benchmark-46-of-168-agent-rollouts
- How this was made: Drafted with Claude, Anthropic's model from 1 cited source, then checked against it in a second editing pass. Editorial policy: https://www.samcodeman.com/editorial-policy

## The short version

- FrontierPhysics grades whole research projects, from literature review to manuscript, instead of fixed questions, isolated sub-tasks or reproduction of papers with given solutions.
- Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts, about 27%.
- Manual review attributes agents' main weaknesses to insufficiently rigorous reasoning and limited expert intuition, not to gaps in physics knowledge or execution errors.
- The abstract does not name the models, the scaffolds, the cost per rollout or the nine domains, so conclusions about agent design are provisional until the full paper is read.

The FrontierPhysics benchmark, posted to arXiv on 2026-11-04, measures whether AI agents can carry out a complete physics research project, from literature review to a written manuscript. It contains 56 tasks made by 42 physicists from their own research. It grades correctness with deterministic verifiers and publishability with expert-authored rubrics. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts, about 27%.

The authors argue that existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions. They add that those benchmarks rarely check research quality alongside correctness. FrontierPhysics is meant to close both gaps. Each task is drawn from a contributing physicist's own research, and each submission is judged on whether the result is correct and whether the research is publishable.

For engineers building research or analysis agents, the headline number is an upper bound on what the tested setups achieve on long-horizon, expert-graded work. The failure analysis may be more useful than the score. It says agents mostly fall short on rigorous reasoning and expert intuition, not on physics knowledge or execution. That is a different problem from one that retrieval or more robust tooling would address.

## What the FrontierPhysics benchmark asks agents to do

Each of the 56 tasks is end-to-end. The abstract lists four stages an agent has to complete:

1. Review the literature.
2. Develop a research plan.
3. Conduct research and experiments.
4. Produce structured outputs and a scientific manuscript.

This differs from question-answer physics evaluations. The agent develops its own research plan, and the output includes a manuscript as well as structured results. The abstract does not describe the agent environment. It does not say which tools agents get, whether literature review runs against live sources or a fixed corpus, what compute or wall-clock budget applies, or how long a typical rollout takes.

Those details determine what a passing rollout costs and how comparable the configurations are. A team planning to run the tasks internally would need the full paper.

## How 56 tasks from 42 physicists span nine domains

The tasks cover nine domains across theoretical, computational and experimental physics. There are more tasks (56) than authors (42), so at least some physicists contributed to more than one task. Because tasks come from the authors' own research, each one is grounded in work the author knows in depth. The rubrics used for grading are also expert-authored.

The abstract does not name the nine domains. It does not say how the 56 tasks split across them or across the three branches. It also does not explain how an agent conducts experiments in experimental physics tasks, for example whether it works from provided datasets, runs simulations, or something else. Per-domain results would show whether the 27% ceiling is spread evenly or concentrated in some areas, and the abstract does not report them.

## Verifiers check correctness, expert rubrics check publishability

Grading has two parts. Deterministic verifiers provide a mechanical check on outputs. Expert-authored rubrics cover research quality, which the authors say existing benchmarks rarely check alongside correctness. The authors summarize the goal this way:

> FrontierPhysics evaluates whether agents' research is correct and publishable.
>
> - FrontierPhysics authors, arXiv 2611.06910v1 abstract

The abstract does not say how the two signals combine into a pass. A rollout might need to clear both the verifier and a rubric threshold, or the scores might be weighted. The abstract also does not say who applies the rubrics at evaluation time, human experts or a model acting as judge, and it reports no agreement figures between graders. Readers need the scoring rule and its reliability before treating small differences between configurations as real.

## No configuration passes more than 46 of 168

The evaluation covers 22 models in 26 configurations, so at least some models were run under more than one setup. Each configuration has 168 rollouts. That is three times 56, which is consistent with three rollouts per task, though the abstract does not state the per-task split. No configuration passed more than 46 rollouts, a rate of 27.4%.

The abstract leaves out most of what a builder would want to know next. It does not name the models or the agent scaffolds behind the 26 configurations. It does not say whether the top passes are spread across many tasks or concentrated in a few tasks solved repeatedly. That is the difference between broad partial competence and a handful of tractable problems. It reports no cost, token usage or runtime per rollout, and reading the leaderboard as a model ranking requires the per-configuration results in the full paper.

## Weak rigor and expert intuition, not knowledge, limit agents

The authors manually reviewed agent trajectories to find out where agents fall short. Their conclusion:

> frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.
>
> - FrontierPhysics authors, arXiv 2611.06910v1 abstract

This finding bears directly on engineering choices. Gaps in physics knowledge would point toward retrieval and better context, and execution errors would point toward more robust tooling and retry logic. The reported weaknesses instead sit in reasoning rigor and judgment, and those are less obviously addressed by harness changes alone.

The abstract does not quantify how often each weakness appears or how many trajectories were reviewed. This is a qualitative finding, not a measured breakdown.

## What a 27% ceiling means for research-agent builders

For teams putting agents on open-ended technical work, in physics or elsewhere, the results suggest some general consequences:

- A correctness check alone does not measure research quality. FrontierPhysics pairs deterministic verifiers with expert rubrics because, the authors say, existing benchmarks rarely check quality alongside correctness.
- The reported weaknesses are in rigorous reasoning and expert intuition. Review should therefore examine the reasoning and the choice of approach, not only whether the code ran.
- No tested configuration passed more than 46 of 168 rollouts. A single agent run on an expert-level research problem should be treated as a draft that needs verification, not as a finished result.
- Internal evaluations of research agents can borrow the structure: tasks written by domain experts from their own work, a deterministic check where one exists, and an expert-authored rubric for research quality.

This is v1 of the paper, and the claims above rest only on its abstract. The details needed to act on the benchmark directly are in the full paper or not yet published: model names, scaffolds, per-domain scores, the pass rule and the cost per rollout. The abstract does establish four things: 56 tasks made by 42 physicists, grading that combines deterministic verifiers with expert rubrics, a top score of no more than 46 of 168 rollouts, and a stated diagnosis that the main weaknesses are reasoning rigor and expert intuition rather than knowledge.

## Questions this raises

### What is the best pass rate on the FrontierPhysics benchmark?

No configuration passes more than 46 of its 168 rollouts, about 27.4%. The abstract does not name the models or scaffolds behind the 26 configurations, so the full paper is needed to see which setup reached that ceiling.

### How does FrontierPhysics grade agent research?

Deterministic verifiers check whether results are correct, and expert-authored rubrics judge whether the research is publishable. The abstract does not say how the two signals combine into a pass or whether humans or a model apply the rubrics.

### Why do agents fail on FrontierPhysics tasks?

From manual review of trajectories, the authors conclude that agents mainly lack rigorous reasoning and expert intuition, not physics knowledge or execution ability. The finding is qualitative, and the abstract does not quantify how often each weakness appears.

## Key facts

- FrontierPhysics was posted to arXiv on 2026-11-04 as v1 of the paper.
- 56 end-to-end research tasks were contributed by 42 physicists from their own research, spanning nine domains.
- 22 models were evaluated in 26 configurations, each with 168 rollouts.
- The best configuration passed 46 of 168 rollouts, a rate of about 27.4%.
- Grading combines deterministic verifiers for correctness with expert-authored rubrics for publishability.

## Sources

1. [FrontierPhysics: Evaluating Agents for End-to-End Physics Research](https://arxiv.org/abs/2611.06910v1) - arXiv