# Parsimony benchmark counts the net code units AI models add to published patches

> Parsimony parses patches that coding benchmarks already publish and scores them by syntax units added minus removed. That gives teams a size signal next to the solve rates from SWE-bench Verified.

- Published: 2026-10-14
- By: Samir Sengupta (https://www.samcodeman.com/about)
- Canonical: https://www.samcodeman.com/writing/parsimony-benchmark-net-code-units-ai-models-add
- How this was made: Drafted with Claude, Anthropic's model from 1 cited source, then checked against it in a second editing pass; read and approved by Samir Sengupta on 2026-10-18. Editorial policy: https://www.samcodeman.com/editorial-policy
- Disclosure: I build Wayne, an AI coding harness for VS Code, and Omnitron, an agent harness, so I work in this category.

## The short version

- Parsimony ranks models by mean net coding units added on solved tasks. Net units are syntax elements added minus syntax elements removed, and lines are not counted.
- The score comes only from patches that benchmarks have already published. Only Python is measured, tests, docs and generated files are excluded, and nothing is executed. Pass/fail comes from SWE-bench's published results.
- Models that write more code also tend to solve more tasks (Spearman 0.33 to 0.60 on boards with 26 or more models), so the lowest-bloat model is not automatically the best choice.
- The project states that units measure size, not readability or design. One commenter on Hacker News said the model they described as winning, Astra, produces hard-to-read code.

Parsimony, an open-source benchmark posted to Show HN on October 13, 2026, measures how much code AI coding models add. It parses patches that benchmarks have already published and counts net coding units: syntax elements added minus syntax elements removed. It then ranks models lowest first by the mean on the tasks they solved. The page text available for this note does not include the leaderboard rows, so it cannot name the most and least bloated models. The two figures it does give matter for model selection: code size correlates with solve rate at Spearman 0.33 to 0.60, and one model's code size changed to 0.88× when run under a different setup.

The stated motivation is narrow. Existing benchmarks measure solve rate and cost but do not punish excess code, which makes a codebase harder to work with in the long term. Parsimony is hosted at parsimony.lasu.dev. For its ranked board it runs no new agent evaluations and instead re-reads diffs that already exist.

## How the Parsimony benchmark counts code units

Each published patch is applied to the task's code, and both the before and after versions of every changed file are parsed. The unit of measurement is a coding unit, meaning a syntax element such as a statement, call, comparison, name or literal. Lines are not counted, so cramming code onto one line earns no credit. Deleted code is subtracted, so a patch that removes code can score lower than one that only adds.

- Net units = added minus removed, computed from the parsed files rather than from line diffs.
- Only Python is measured.
- Tests, docs and generated files are left out, so writing tests does not raise a model's count.
- Nothing is run. Pass or fail comes from SWE-bench's published results, and tests are not rerun.

One commenter on Hacker News said that measuring at the abstract syntax level removes the layer of mess that comments and formatting add. The tradeoff is that Parsimony relies on SWE-bench's published pass/fail labels, because it does not rerun the tests itself.

## Which models rank most and least bloated

The leaderboard sorts models by mean net units added on the tasks they solved, lowest first. A second column, net units added (mean), also includes failed attempts. Solved is taken from SWE-bench Verified, and Measured reports how many solved attempts could be measured. The table also lists an entry labelled Maintainers' pull request, and a task selector defaults to all tasks.

The Artificial Analysis Intelligence Index appears alongside each model as an outside score, matched by model and reasoning effort. It does not affect the rank.

The source text does not include the actual rows, so this note cannot give an ordering from the source. One commenter on Hacker News said Astra is "winning" the benchmark, and contrasted it with what they called the excessive comments and tests of Claude. Under Parsimony's method, comments are not coding units and tests are excluded, so that particular complaint would not show up in the score. Treat the commenter's ordering as a practitioner report and check the live table before relying on it.

Some models on the page are shown faded. The author ran these on the first 3 to 10 DeepSWE Python tasks, using pi for OpenAI models and Claude Code for Anthropic models, and graded them with each task's own tests. They are not verified or ranked.

GPT-5.6 Sol was run both ways, and the author's run added 0.88× the code of its published run. The two runs differ in harness, in task set and in grading, so the source does not show how much of that gap comes from the harness alone.

## Fewer units is not the same as readable code

The project's own example shows what the metric rewards. The page says these two functions return the same result, and the second has fewer coding units.

```python
def has_negative(values):
    found = False
    for value in values:
        if value < 0:
            found = True
    return found

def has_negative(values):
    return any(value < 0 for value in values)
```

In this example the shorter version is also the clearer one. The main objection on Hacker News is that this does not always hold. One commenter said Astra plays code golf and produces code that is hard to read, and that they would rather accept Claude's verbosity. Another said that code that is too verbose is bad, and so is code that is too terse.

Parsimony states the same limit directly. It offers no readability metric. One commenter asked for cyclomatic complexity as a supporting metric for readability, and the sources do not say whether the project plans to add it.

> Smaller isn't always better: units measure size, not readability or design.
>
> - Parsimony, Limits section

## Code size correlates with solve rate (Spearman 0.33–0.60)

For model selection, the most important caveat is in the limits section. Models that write more code also tend to solve more tasks, with a Spearman correlation of 0.33 to 0.60 on the boards with 26 or more models. Picking the lowest-ranked model on Parsimony alone could mean giving up solve rate. The page's chart lets you plot two metrics against each other on a scatter, so you can view size and Solved together.

The sources do not say whether the extra units in higher-solving models are necessary work or padding. They also do not answer one commenter's question: does less code actually mean less work for whoever reviews it? The benchmark measures diff size, not review time.

## mini-SWE-agent fixes the harness, not its influence

Every ranked model runs through the same agent, mini-SWE-agent, so differences between rows do not come from different scaffolds. The project still notes that the agent affects how much code gets written. A model's Parsimony number therefore describes that model inside mini-SWE-agent, not inside Claude Code, pi or another harness. The GPT-5.6 Sol comparison is the only cross-setup figure on the page, and it changes task set and grading as well as harness, so it does not measure the harness effect on its own.

## When Parsimony is sound enough to choose a model

The method is static, and the page lists its data sources and hashes. It also lets anyone load their own results: download the example JSON, fill in code-size numbers from another benchmark, and open it to get the same chart and table. The file stays in the browser. For teams that already run internal evals, this is the most practical use, because it applies the same size accounting to their own tasks and harness instead of the published patches.

- Use it as a tiebreaker between models with similar solve rates, not as a primary ranking.
- Check the Measured column, since a mean over few measured attempts is weak evidence.
- Results cover Python only, so non-Python codebases are not represented.
- Pair it with human review of readability, since one commenter reported readability problems with the model they described as winning.

The sources also say nothing about coverage of small open-weight models. One commenter asked for benchmarks like this to include locally runnable models such as Qwen 3.8 27B, and the source does not say whether Parsimony's board covers them. Parsimony gives a count of how much code a model adds. Whether that added code is worth its cost is still a judgment the reviewer has to make.

## Questions this raises

### How does the Parsimony benchmark measure code size?

It applies each published patch, parses the before and after versions of every changed file, and counts net coding units, meaning syntax elements added minus syntax elements removed. Lines are not counted, and deleted code is subtracted, so a patch that removes code can score lower than one that only adds.

### Should I pick the model with the lowest Parsimony score?

Not on that alone. Models that write more code also tend to solve more tasks, with a Spearman correlation of 0.33 to 0.60, so the lowest-ranked model may give up solve rate. The project also offers no readability metric, so use it as a tiebreaker between models with similar solve rates.

### Does Parsimony rerun the SWE-bench tests?

No. Nothing is run for the ranked board, and pass or fail comes from SWE-bench Verified's published results. Every ranked model uses the same agent, mini-SWE-agent, so the scores describe models inside that harness.

## Key facts

- Parsimony was posted to Show HN on October 13, 2026 and is hosted at parsimony.lasu.dev.
- Code size correlates with solve rate at Spearman 0.33 to 0.60 on boards with 26 or more models.
- GPT-5.6 Sol added 0.88x the code of its published run when the author ran it under a different setup.
- Unranked, faded models were run on the first 3 to 10 DeepSWE Python tasks using pi or Claude Code.
- Only Python is measured, and tests, docs and generated files are excluded from the count.

## Sources

1. [Show HN: Parsimony – an open-source benchmark for how much code AI models add](https://parsimony.lasu.dev/) - parsimony.lasu.dev

Discussion read (practitioners' reactions, not a source): https://news.ycombinator.com/item?id=50077619