STEPQuant keeps delta-rule recurrent states near FP32 accuracy at a 6-bit budget

The problem is specific to linear attention. Instead of a KV cache that grows with every token, these models keep a fixed-size recurrent state. That state persists, and under concurrent serving the authors say these states "can become a substantial memory bottleneck." STEPQuant is an argument about where in that state, and over what span of decoding, low precision is safe.

The recurrent state is updated at every decoding step, and the paper's central point is that quantization error does not stay local to the step that introduced it. It propagates through successive state updates. If the state is stored in low precision, later steps build on a state that already carries earlier rounding error.

This is the baseline the paper reacts to: direct quantization of the recurrent state to low precision. The abstract calls the result severe accuracy degradation but gives no figure in the available text. For a serving team, the consequence is that state compression cannot be treated as a single bit-width setting. Because the error propagates through updates, the precision decision has to account for time.

The first observation is temporal. The authors write that errors in long-lived memory can persist across many decoding steps. On a plain reading, information the delta rule keeps in the state for a long time carries any quantization error with it for that whole period, while information that is soon overwritten does not. The framework accordingly allocates precision according to error magnitude and memory lifetime. The abstract does not spell out the direction or size of that allocation, though the observation implies long-lived memory needs more protection.

For serving, the open question is which update steps can safely run at low precision. The abstract states the principle of allocating by error magnitude and memory lifetime. It does not give the concrete schedule: which steps, which state regions, and at what bit-widths. Implementing the allocation rule will require the full paper.

The second observation is spatial, and the paper reports two separate facts about it. First, errors in different key rows affect model outputs differently, so some rows matter more to the final prediction than others. Second, state magnitudes vary substantially along both rows and columns, which means one scale for the whole state would fit the distribution poorly.

Fitting scales on both axes at once addresses magnitude variation in two directions. Basing the fit on key-row impact on output error, as well as on the state distribution, means the scales account for the error the model's outputs see, not only how the state values are spread.

The paper describes STEPQuant as a spatial-temporal post-training quantization framework, and the precision decision has two inputs. The temporal input allocates precision by error magnitude and memory lifetime. The spatial input sets scales from row and column magnitudes and per-row output impact. Because it is post-training, the method is applied to existing checkpoints such as Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct rather than requiring a training run. That matters for teams serving models they did not train.

The headline figure is a nominal 6-bit budget. The word nominal, combined with lifetime-based allocation, suggests an average across a mix of precisions rather than a uniform 6 bits everywhere. The abstract does not break down that mix or say how the scales themselves are stored.

Whether this is worth integrating depends on how many more concurrent sequences fit on a GPU, and the abstract does not answer that. What can be derived is a ceiling. The reference point is FP32 state, and 6 bits is 18.75 percent of 32 bits, so the raw state payload would shrink by a factor of about 5.3. That is arithmetic on the stated numbers, not a measured result from the paper.

Several factors the abstract does not quantify could pull the real figure below that ceiling. The fitted key-row and value-column scales are metadata that must be stored with the state. Whether the state is held in a packed 6-bit layout or padded to a byte-aligned format also affects the result. Capacity planning should wait for measured numbers.

The experiments cover two models, Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct, each evaluated on long-generation and short-generation benchmarks. The split matters given the paper's thesis. If errors persist across many decoding steps, long generations are where a weak scheme should fail, so parity on both is the relevant test.

The abstract says STEPQuant closely matches FP32-state accuracy under the nominal 6-bit budget. The available text is truncated at "outperforms u" before it names the baselines or the margins. The size of the gap over naive state quantization therefore cannot be stated from this source.

The paper is a v1 posted on 29 September 2026, and the abstract leaves the engineering numbers open. Before a serving stack depends on this approach, the following need to be settled by the full text or follow-up work.