Jevstiller's disagreement bound: what distilling Jev into a local model guarantees

A Jev classification call takes about 300 ms at any load. For an agent loop, a game tick or anything that classifies and then acts, that is the whole budget. Jevstiller sits in front of those calls as a proxy, released under Apache 2.0. It embeds each request with a frozen bge-small encoder (384 dimensions, ONNX Runtime on CPU). On top of that it trains a multinomial logistic regression head with full-batch Adam and early stopping, by cross-entropy against Jev's full probability distribution over the labels.

The head is a few hundred kilobytes and trains in seconds on a few thousand rows. It is retrained every 2,000 new Jev answers, versioned, and shadow-tested on live traffic before promotion. A request above the confidence threshold and inside the training distribution, judged by a k-nearest-neighbour out-of-distribution scorer, gets a local answer in about 15 ms. Everything else goes to Jev.

Per task, let c be coverage, the share of requests the local model answers. Let e be the share of those where its label differs from Jev's. Forwarded requests agree with Jev by definition, so system agreement is A = 1 − c·e. A target A* sets a budget β = 1 − A*, which is 2 in 100 at 98%, and the router answers as much as it can while keeping c·e under β.

Two things are deliberately left out. First, the bound says nothing about accuracy against the truth: if Jev is wrong, the local model is wrong the same way. Second, it is not a claim about the local model's accuracy on the requests it chose to answer, which the write-up says is what most tools report. It is a statement about disagreement over all traffic, which can be verified against Jev directly.

The usual recipe holds out data, sweeps a confidence threshold, and ships the loosest one whose measured disagreement is within budget. Measured disagreement at any threshold is a noisy estimate. The loosest threshold that looks under budget is therefore preferentially one whose noise pointed downward, and on new traffic it can deliver 2.7% where it measured 1.9%. The benchmark compared that rule against the bound across twenty train/calibration/test splits per task, and the point estimate overshot by up to a full percentage point.

The bound is a 95% statement, so about 5 in 100 splits may miss, and one miss in 100 is within that. Each retrain spends the 5% again, which is why promotion-time checks are not enough on their own. The write-up does not state a minimum number of calibration rows or the size of the IID calibration split. It says only that the head trains on a few thousand rows and that the status report is worth reading after a few thousand requests. A prior-art review found two errors in the first version: it bounded the selective rate while treating estimated coverage as exact, and it kept the loosest of 200 individually passing thresholds; fixing both moved Banking77 replay coverage only from 70.6% to 70.7%.

Coverage follows Jev's consistency, not the task's difficulty. On the two tweet tasks, Jev agrees with human labels only 64% and 74% of the time. The local model cannot reproduce that noise within 2%, so under the bound it answered 24.0% and 28.7% locally, against 74.9% to 86.5% on the intent and news tasks. Training on Jev's full probability distributions rather than its top label buys back two to three points on the many-class tasks. Before swapping in, check these points:

One commenter asked whether this is a near drop-in that shrinks a Jev bill over time. The author's opening post notes that it speaks Jev's API only, with an OpenAI-compatible front on the roadmap. Deployment is a Docker container that TYPESAFE_BASE_URL points at. Another commenter argued it only works for simple text classification. A reply countered that it targets questions asked many times, and that the replier's own Jev questions repeat thousands of times.

One commenter asked what type of head this is. The write-up answers that: a multinomial logistic regression layer on frozen bge-small embeddings. The same commenter also wanted a way to correct Jev's wrong answers. Nothing in the write-up describes a correction path, and the contract as written reproduces Jev, errors included. Another commenter contrasted it with model2vec, which distills a sentence transformer into a faster general-purpose encoder, whereas Jevstiller keeps the encoder frozen and distills Jev's decisions on one question into a head.

The benchmarks rerun without an API key, from Jev's recorded answers:

The evidence covers five public classification tasks and one simulated teacher change in a 24-hour soak. The sources give no minimum calibration size per task and no picture of how the bound tightens as rows accumulate. They report no results on private or multi-turn workloads, and per-class guarantees do not exist yet. The bound assumes calibration rows are a random sample of the traffic they gate, so a shifted mix is left to the audit to catch. The status report prints both the promotion-time bound and the audit's measured interval, and those are the two numbers to watch.