the short version
- Softmax Collapse: standard implementations round the target logit's gradient to zero when that logit is much larger than the others, so the gradients no longer sum to zero.
- Softmax Cancellation: when student logits are close to teacher logits, standard implementations produce nonzero gradients, even when the two are equal.
- The authors report that their fixed implementations eliminate the Slingshot phenomenon, speed up grokking with weight decay, prevent accuracy collapse without weight decay, and stop students drifting away from the teacher in distillation.
- The paper says that after memorization, logit growth is bounded by an equilibrium between weight decay and Adam's epsilon hyperparameter.
Standard implementations of softmax, log-softmax and cross-entropy contain two numerical bugs, according to "A Fix for Softmax and Cross-Entropy" (arXiv 2610.31642v1, posted 2026-10-22). Softmax Collapse rounds the target logit's gradient to zero when that one logit is much larger than the others, so the gradients no longer sum to zero. Softmax Cancellation produces nonzero gradients when student logits are close to teacher logits, including when the two are exactly equal. The authors report that their replacement implementations eliminate the "Slingshot" phenomenon, speed up grokking with weight decay, prevent accuracy collapse without weight decay, and stop students drifting away from the teacher during distillation. The abstract gives no numeric threshold for "much larger" and no magnitude for the spurious gradients.
The abstract describes both bugs as wrong gradient values, not as failures. One returns a zero where a gradient should be. The other returns a gradient where there should be zero. The trigger conditions are a single logit far above the rest of its row, and student logits close to or equal to a teacher's.
The authors present replacement implementations of log-softmax, softmax and cross-entropy that fix both bugs. The abstract claims the fix changes several training behaviors:
- It eliminates the previously observed "Slingshot" phenomenon.
- It speeds up grokking when weight decay is used.
- It prevents accuracy collapse when weight decay is not used.
- It stops student networks from drifting away from the teacher during distillation.
- It shows that after memorization, logit growth is bounded by an equilibrium between weight decay and Adam's epsilon hyperparameter.
When Softmax Collapse zeroes the target gradient
The first bug appears when one logit, the target, is much larger than every other logit in the row. In that case the standard implementation rounds the target logit's gradient to zero. The abstract states the result as gradients that no longer sum to zero, and presents that as part of the bug.
The first bug, Softmax Collapse, rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero.
The trigger is one dominant logit. A plausible reading is that this is the state of a network that has become very confident on its training examples. That reading would explain why the paper's claims about grokking, accuracy collapse and post-memorization logit growth sit alongside this bug, but the abstract does not draw that link explicitly.
The abstract leaves several practical questions open. It does not say how large the gap between logits must be before rounding occurs. It does not say which floating-point formats the authors tested, whether fp32, bf16 or fp16 behave differently, or which frameworks' built-in functions show the bug. Engineers will need the full paper, or a direct test, before assuming a particular stack is exposed.
Why Softmax Cancellation breaks distillation at equality
The second bug matters when a loss compares two sets of logits, as in distillation. When the student's logits are close to the teacher's, the standard implementation produces nonzero gradients. It does so even when the two sets are exactly equal, which the paper treats as a bug. At equality the student already matches the teacher on that example, so any gradient there is spurious.
The paper reports that its fix stops student networks from drifting away from the teacher during distillation. The implication is that the spurious gradient near equality is what moves an already-matched student off target. The abstract gives no figures on how far students drift under the standard code, on which model pairs, or after how many steps. It also does not say whether temperature-scaled distillation losses are affected differently.
Slingshot and grokking change once both bugs are fixed
The abstract says the fix eliminates the "Slingshot" phenomenon, which it describes as previously observed. It does not call Slingshot an artifact. If a fix to softmax numerics makes the behavior disappear, however, it is reasonable to infer that at least some Slingshot observations depended on the numerical error. Work that builds on Slingshot results would need to account for that.
The paper makes two claims about grokking. With weight decay, the fixed implementation speeds up grokking. Without weight decay, it prevents accuracy collapse. The abstract does not report the size of the speedup, the tasks or architectures used, or whether the effects were measured beyond the experimental setups in the paper.
Adam's epsilon and weight decay bound logit growth
The final claim concerns what happens to logits after memorization. According to the abstract, logit growth is then bounded by an equilibrium between weight decay and Adam's epsilon hyperparameter. That makes epsilon one of two settings that determine where logits settle late in training.
If epsilon helps set where logits settle, then changing it can alter logit magnitude in runs that continue past memorization. That may be relevant to the first bug, since Softmax Collapse is triggered by one logit being much larger than the others. The abstract does not make that connection directly. It also does not give the equilibrium formula, the epsilon values tested, or whether the result applies to Adam variants.
What the fix changes in a training loop
The authors describe their implementations as simple, and the implementations cover log-softmax, softmax and cross-entropy. The abstract does not include the code, name a package, or state compute or memory overhead. Based on what it does say, these are reasonable steps for a team that trains confident classifiers or runs distillation:
- Write a unit test for the distillation loss that passes identical student and teacher logits and asserts the gradient is exactly zero. A nonzero result indicates Softmax Cancellation in the current implementation.
- Write a test that passes a row where one target logit is much larger than the rest. Check whether the target gradient is zero and whether the logit gradients sum to zero.
- If either test fails, replace log-softmax, softmax and cross-entropy with the implementations the paper presents, then rerun both tests.
- Re-examine conclusions drawn from Slingshot-like behavior, accuracy collapse without weight decay, or student drift in distillation, since the paper reports that its fix changes all three.
- Treat Adam's epsilon as a setting that, together with weight decay, bounds logit growth after memorization.
What is known so far is limited. The paper is version 1, posted on 2026-10-22, and the only detail available here is the abstract. The abstract contains no quantitative results and names no frameworks. The bug descriptions are specific enough to test against an existing stack now. Whether the fix changes final model quality at production scale is a question the abstract does not answer.
Questions this raises
What is Softmax Collapse?
Softmax Collapse is a numerical bug in standard softmax, log-softmax and cross-entropy implementations. When one target logit is much larger than the others, its gradient is rounded to zero, so the gradients no longer sum to zero.
How can I test whether my training stack has these bugs?
Pass identical student and teacher logits to your distillation loss and assert the gradient is exactly zero. Then pass a row where one target logit is much larger than the rest and check whether the target gradient is zero and whether the gradients sum to zero.
What does the fix change in training behavior?
The authors report that their implementations eliminate the Slingshot phenomenon, speed up grokking with weight decay, and prevent accuracy collapse without weight decay. They also stop student networks drifting from the teacher during distillation, though the abstract gives no quantitative results.
Spotted an error? Email samir.s.sengupta@gmail.com and I will correct it and date the change at the top of the note. How these notes are researched, drafted and checked is in the editorial policy.