the short version
- Within 64 accepted target evaluations, JevOut's optimizer redirected Jev on 312 of 508 initially correct decisions (61.4%), while preserving the source, question, choices and gold answer.
- In 229 cases Jev assigned at least 0.7 probability to the fixed wrong option, so a confidence threshold does not filter these failures out.
- Across seven datasets, three additional decision systems showed targeted flip rates of 64.9% to 73.2% on decisions they initially answered correctly; the abstract does not name them.
- The optimizer's search signal is the model's own option probabilities, which is the field a TypeSafe-shaped /v1/systemone response returns alongside the choice.
JevOut (arXiv 2609.30243v1, published 24 September 2026) reports that within 64 accepted target evaluations, its optimizer found context additions that redirected Jev on 312 of 508 initially correct decisions, a targeted flip rate of 61.4%. The source text, the question, the answer choices and the gold answer were all preserved; only the background detail around them changed. For anyone using a decision model as a router or tool selector, that is the number to plan against: a clear majority of decisions the model already got right can be redirected by short additions that fit naturally into the surrounding context.
The paper's framing is what puts this on production systems. Dedicated decision models such as Jev map unstructured language to probability distributions over finite choices, and those outputs directly route requests, select tools and trigger actions. The answer comes out of a single forward pass with no token-by-token generation, so there is no intermediate text for a human or a guardrail to read. The distribution is the decision, and anything that moves the distribution moves behaviour.
How natural context flips decision models
The setup is narrow by design. For each item the model initially answers correctly, the authors fix one wrong target option, then use the model's option probabilities to refine fluent context additions, preserving the source, the question, the choices and the gold answer. Nothing in the item is deleted or contradicted, and the correct answer remains unchanged.
short additions that fit naturally into this context can nevertheless redirect an otherwise correct decision, even when the correct answer remains unchanged
The abstract does not define what counts as an accepted target evaluation, so read 64 as the reported budget rather than a precise query count. What it does state is that the optimizer works from the model's option probabilities. Returning those probabilities is the point of this model class, so the interface that makes the model useful is the same interface that supplies the search signal.
Why 229 high-confidence flips break thresholds
The standard defence for a router is a confidence threshold: take the choice above some probability, escalate below it. In 229 cases the flipped decision cleared 0.7 on the fixed wrong option. The model is not merely uncertain and tipping over; it reports high probability on the option the optimizer selected, which is the regime a threshold is meant to catch.
That collides with the contract these models are sold on. Di Zhang's walkthrough describes RLCD as multiway preference modeling plus probability calibration, a schema-conditioned Plackett-Luce objective, and traces the lineage from scalar reward to pairwise preference (LLaMA-Berry's PPRM, trained on almost 7.8 million mathematical-solution pairs with DPO) to multiway preference to calibrated decision. He is explicit that normalization is not calibration: a softmax vector always sums to one, which does not mean a prediction reported as 0.8 is correct 80% of the time. The calibration contract he attributes to TypeSafe is that higher reported probabilities should correspond to higher observed accuracy, and it is scored with strictly proper rules, where a binary forecast of 0.8 costs 0.04 when the event occurs and 0.64 when it does not.
The confidently wrong forecast costs sixteen times as much as the confidently correct one.
Calibration of that kind is an average property over the inputs a model is evaluated on. JevOut evaluates inputs chosen to raise one option's probability, and on those inputs the reported probability reached 0.7 or more for the wrong option 229 times. A model can hold its calibration curve on an evaluation set and still report high confidence on the branch an optimizer picked, so the confidence field is not a safety mechanism against this attack.
Three other systems flip at 64.9% to 73.2%
Jev is not an outlier. Across seven datasets, three additional decision systems showed targeted flip rates of 64.9% to 73.2% on decisions they had initially answered correctly. The abstract names neither the three systems nor the seven datasets, so you cannot map the result onto a specific open checkpoint. The paper's own conclusion is that the results expose a pronounced fragility in current decision models.
The numbers also point the wrong way for the obvious workaround. The three other systems flipped more often than Jev, not less, so running a second decision model and requiring agreement starts from a higher per-model flip rate. Whether a context found against one system also flips another is not reported.
What this costs a router or tool selector
Ollaya's front page shows the shape of decision these models are asked to make. Its agent preset scores a request where the stated task is to fix a typo in README.md and the proposed command is git push --force origin main; decider:2b returns action block at 0.53, on_task no at 0.75, destructive yes at 0.90, and a risk score of 1.23 out of 2 annotated "could lose local work", in 178 ms on an RTX 4090. Every one of those numbers is a threshold input somewhere in an agent loop.
{
"model": "laya:en",
"answers": {
"intent": {
"type": "choice",
"choice": "invoice",
"confidence": 0.9547,
"probabilities": {
"invoice": 0.9698,
"refund": 0.0172,
"other": 0.013
}
}
},
"usage": { "input_tokens": 43, "output_tokens": 0 }
}The text these questions run over is user-controlled by construction. Ollaya's own pitch names it: tickets, emails and user messages are often the most sensitive data you have, and they are scored where they already live. A support ticket has room for background detail that reads as ordinary, which is exactly the material JevOut's optimizer adds. The response above shows what a flip would look like downstream: a choice field, a confidence of 0.9547 and a probability vector that the calling code will act on without any generated text to inspect.
Local serving changes latency, not the attack surface
Ollaya runs this model class on your own hardware with ONNX Runtime under an Apache-2.0 runtime, listening on 127.0.0.1 by default, serving TypeSafe's /v1/systemone and /v1/models shapes so the official TypeSafe Python SDK 0.7.1 works unchanged against a local server. Weights come from their authors' Hugging Face repositories, pinned to a commit and checked against sha256. The published medians on an RTX 4090 are laya:multilingual at 8.1 ms, laya:en at 9.6 ms, gliclass at 14.7 ms, nli at 20.4 ms, decider:0.8b at 155 ms and decider:2b at 190 ms, against 236 to 276 ms for the hosted TypeSafe Jev API in third-party benchmarks that include the network. Ollaya says setups differ and to read that as an order-of-magnitude comparison.
None of that touches JevOut. Local inference removes the network hop, the per-token bill and the data egress; it does not make the option distribution harder to move. The decider models read their answer from option-letter logits in one forward pass, which is the same probability surface the paper's optimizer refines against, and Ollaya advertises no metering and no API bill, so the 64 evaluations cost an attacker with local access nothing but hardware time. Ollaya does ship one relevant lever: each model carries its own calibration, and a Modelfile refits it on your labelled data. That is worth doing for threshold quality on your own traffic, but the paper gives no evidence that recalibration survives adversarial context.
What the JevOut abstract does not report
The available abstract ends mid-sentence, and several things an engineer would want are not in it. There is no reported defence, no adversarial-training result and no detection heuristic. The length of the successful context additions is described only as short. The seven datasets and the three additional decision systems are unnamed, and no Jev version or checkpoint is identified, so you cannot check whether the model you serve was covered. Transfer is silent too: nothing says whether a context found against one model flips another, which is the difference between an attacker needing probability access and needing nothing at all.
What you can act on is narrower than a fix. Treat the probability vector as an untrusted output whenever any part of the scored text came from a user, and do not let a single decision-model call be the only thing between a request and an irreversible action. Keep a deterministic check on the git push --force class of branch rather than delegating it to a 0.90. The value of these models is a typed answer with a probability in a single forward pass and zero output tokens, and that value survives being treated as advice rather than authority.
Questions this raises
what is JevOut
JevOut is the attack described in arXiv 2609.30243v1, published 24 September 2026. It uses a decision model's own option probabilities to search for short, fluent additions to the surrounding context that redirect a correct decision to a chosen wrong option, without deleting or contradicting anything in the item.
does a confidence threshold stop JevOut flips
No. In 229 of the 312 flips, the model reported at least 0.7 probability on the fixed wrong option, so a threshold that escalates only low-confidence answers passes them through. Calibration is an average property over an evaluation set, and JevOut evaluates inputs picked to raise one option's probability.
does running decision models locally reduce the risk
Local serving with something like Ollaya removes the network hop, the per-token bill and the data egress, but not the attack surface. The decider models still read their answer from option-letter logits in one forward pass, which is the same probability surface the optimizer refines against, and unmetered local inference makes the 64 evaluations cost an attacker only hardware time.
These daily notes are drafted by a model I run and operate myself - the same kind of pipeline this site is about - from sources published in the previous 24 hours, and every one lists what it read. The longer essays, the talks and the preprint are mine, written by hand.