# Dayjob tax benchmark: the best AI agent passes 10.8% of 50 tax assignments

> Surge AI's DAYJOB: Tax gives agents 50 underspecified professional tax assignments. The top model, Claude Sonnet 5.5, passes 10.8%, and Surge's case studies show agents missing issues the client never raised.

- Published: 2026-10-28
- By: Samir Sengupta (https://www.samcodeman.com/about)
- Canonical: https://www.samcodeman.com/writing/dayjob-tax-benchmark-best-agent-passes-10-8-percent
- How this was made: Drafted with Claude, Anthropic's model from 1 cited source, then checked against it in a second editing pass. Editorial policy: https://www.samcodeman.com/editorial-policy

## The short version

- On the DAYJOB: Tax benchmark, the leading agent, Claude Sonnet 5.5, passes 10.8% of 50 professional tax assignments at a 95% rubric threshold.
- Both failures Surge documents are judgment failures. In one, an agent missed a deadline the client never asked about. In the other, an agent built advice on an unverified marital status.
- The top three Claude models sit within 1.6 points of each other, and the post reports no confidence intervals to show whether that ordering is meaningful.
- Teams evaluating agents on professional work should grade whether the agent found the right problem, not only whether it solved the stated one.

Surge AI's dayjob tax benchmark, DAYJOB: Tax, gives AI agents 50 professional tax assignments built from 1,789 files. It grades them pass@1 at a 95% rubric threshold. The best result is Claude Sonnet 5.5 at a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. In both case studies Surge publishes, the agents answer the question they were asked and miss a consequential issue that is sitting in the files.

DAYJOB: Tax extends Surge AI's DAYJOB benchmark family, whose earlier releases covered Healthcare and Finance, into tax preparation, compliance and advisory work. Each assignment starts with a client request and a workspace of supporting records. The agent has to produce a finished deliverable, such as a tax return, workpapers, a technical review or client advice. Surge says the assignments involve decisions that are not fully specified in the initial instructions, and records that conflict. The agent is expected to take ownership of the work from the initial client request to the finished deliverable.

## How the dayjob tax benchmark grades agents

Each task has a rubric, and an attempt passes only if it satisfies at least 95% of that task's criteria. The criteria cover calculations, supporting evidence, consequential issues and the completed deliverable. A correct return that leaves a consequential issue unflagged can therefore still fail. Surge reports pass@1 at that threshold. The 50 environments span eight areas of practice:

- Individual and family tax
- Small-business tax
- Partnerships
- Property and investments
- Estates, gifts, and trusts
- International tax
- Corporate transactions
- Tax review and disputes

Tax experts build the tasks from real-world workflows. They write the supporting records, expected outcomes and grading criteria, and each task goes through a three-layer expert review for correctness, completeness and solvability. Surge describes three contributors. One is a tax attorney with more than 10 years at PwC, Deloitte and Exxon. Another is a Baker Tilly Tax Director with experience at PwC, Moss Adams and Windes. The third is a tax professional from Deloitte and EY who specializes in real-estate partnerships, individual taxation and trusts. The post does not publish the rubrics, the number of criteria per task, or any measure of grader agreement.

## Workspaces hold 1,789 files, mostly PDFs

The 1,789 workspace files include 1,487 PDFs, 117 Excel workbooks and 38 Word documents. The rest are emails, images, CSVs, JSON records and text files. The median assignment has 30 files and the largest has 123. The standard documents are there: filed and draft returns, K-1s, W-2s, IRS notices, general ledgers, payroll records, contracts and correspondence. The workspaces also contain the less standardized evidence that tax professionals routinely work with.

One task includes a photograph of a handwritten receipt for merchandise bought for resale. The agent has to read the item descriptions and amounts, reconcile them with spending records, and work out the effect on inventory and cost of goods sold. In another, the client assumes an industrial silo-services company operates only in Alabama. Its field dispatch log, however, shows crews working across five states, which bears on its multistate tax obligations. A third requires reconciling Polish employment and property records with cryptocurrency exchange exports, wallet histories and historical purchase lots. In each case the agent has to read and reconcile messy evidence before it can apply any tax treatment.

## Agents miss the issue the client never raised

In the first case study, Anna moved out of her condo in August 2024. She asked whether closing in October 2026 or January 2027 would be better for taxes. An agent correctly found that both dates satisfy the two-of-five-years residence requirement for an exclusion of up to $250,000 in gains. It did not tell her that she stops meeting that requirement around the end of August 2027, so a sale delayed past then could lose eligibility. Across five attempts, Astra never identified the deadline, while Opus identified it in all five.

The second case reverses the result. A separating couple asked how to divide a Roth IRA and a 401(k). Opus 5.5 told them they were not legally married and recommended keeping the accounts separate, with a cash payment to even out the balances. They now lived in Florida, where new common-law marriages generally are not recognized. Their records showed, however, that they had lived together in Texas, which recognizes informal marriages under certain conditions, and had held a ceremony in Colorado. Across all five attempts, Opus based its recommendations on the unmarried assumption, while Astra recognized the need to verify marital status in four of five.

> We first need to confirm your legal marital status.
>
> - Astra response quoted in Surge AI's DAYJOB: Tax post

In these two cases, neither model is reliably better. Each caught the issue the other missed, and each behaved the same way in all or nearly all of its five attempts. If that pattern holds more widely, retrying an agent will not fix this kind of error. Surge's conclusion is that even when agents can perform the individual steps, exercising professional judgment and reliably carrying out real-world tax workflows remain out of reach.

## Sonnet 5.5's 10.8% lead is narrow

One commenter on Hacker News was surprised that Sonnet 5.5 did better than Opus and Fable, and said the same pattern shows up in Surge's other benchmarks. Another replied that the difference is within the benchmark's error bars. The post itself reports no confidence intervals. With 50 environments, a single environment equals 2 points, and the spread from first to third place is 1.6 points. The defensible reading is that the three Claude models land in the same band around 10%, not that their order is settled.

Surge also plots pass rate against average cost per trial and places Sonnet 5.5 and Opus 5.5 at the high-performance end of the measured Pareto frontier. The post gives no dollar figures, token counts or latencies. The full model list is on the leaderboard page, and the post links a Hugging Face page. Access to the full benchmark, however, is by request to benchmarks@surgehq.ai. That limits independent verification, which one commenter asked about directly.

## What TaxBench, TaxCalcBench and TaxLab leave out

Surge compares DAYJOB with three existing evaluations. TaxBench asks 114 tax questions across multiple specialties, each with a defined problem and grading rubric. TaxCalcBench checks tax-return accuracy against expected filing values. TaxLab tests whether an agent can question a simulated taxpayer, obtain missing information and complete a federal individual return. Surge's point is that these evaluations begin with a defined problem.

DAYJOB adds two questions: whether the client asked the right question, and whether the records support the tax treatment. In one example, a studio's ledger refers to an attached receipt that is missing. The agent should flag the missing evidence instead of treating the reimbursement as verified. The post does not report how the same models score on TaxBench, TaxCalcBench or TaxLab. It therefore cannot separate how much of the low pass rate comes from problem discovery and how much from harder underlying tax law.

## Grade agents on the question nobody asked

For teams evaluating agents on professional workflows, the benchmark suggests a rubric design. Rubrics should include criteria for consequential issues the prompt never mentions, and workspaces should contain the evidence that ought to trigger them. The 95% threshold matters too. At a looser cutoff, an agent that handles both of Anna's proposed dates correctly but misses the August 2027 deadline could still pass. Running five attempts per task, as Surge's case studies do, separates consistent blind spots from one-off misses.

Commenters on Hacker News describe the same partial competence the benchmark shows. Several reported models finding real errors in their returns or records, including an incorrect exchange rate. One commenter also described highly inaccurate estimated-tax calculations and a model that disagreed with IRS guidance until shown links to it. Another said a model did better than their specialist accountant, but only with a lot of steering. Another had Claude find mistakes, then hired a professional to confirm them and file the amendment. Others raised concerns the benchmark does not address: who takes liability for mistakes, and whether clients want their financial records sent to model providers. A 10.8% top pass rate supports the pattern those commenters describe, where the model acts as a second reviewer and a human signs off, rather than an agent doing the work end to end.

## Questions this raises

### Which AI model scores highest on the DAYJOB: Tax benchmark?

Claude Sonnet 5.5 has the top pass rate at 10.8%, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. Surge reports no confidence intervals, and one task is worth 2 points, so the three models effectively sit in the same band around 10%.

### How does DAYJOB: Tax grade agent attempts?

Each task has a rubric that covers calculations, supporting evidence, consequential issues and the finished deliverable. An attempt passes only if it satisfies at least 95% of the criteria, so a correct return that leaves a consequential issue unflagged can still fail. Results are reported as pass@1.

### How is DAYJOB: Tax different from TaxBench, TaxCalcBench and TaxLab?

Those evaluations begin with a defined problem: tax questions, return accuracy checks, or a simulated taxpayer interview. DAYJOB also tests whether the client asked the right question and whether the records actually support the tax treatment.

## Key facts

- Claude Sonnet 5.5 leads at 10.8% pass@1, ahead of Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%.
- An attempt passes only if it meets at least 95% of the task's rubric criteria.
- The 50 assignments use 1,789 workspace files, including 1,487 PDFs, 117 Excel workbooks and 38 Word documents.
- The median assignment has 30 files and the largest has 123.
- With 50 environments, one task equals 2 points, so the 1.6-point spread from first to third place is not a settled ranking.

## Sources

1. [Dayjob: Tax – evaluating AI agents on 50 professional tax assignments](https://surgehq.ai/blog/dayjob-tax) - surgehq.ai

Discussion read (practitioners' reactions, not a source): https://news.ycombinator.com/item?id=50275626