vLLM speculative decoding on AMD ROCm: five draft methods, no single speedup number

N-gram drafting is not among the five, so the post carries no ROCm evidence for or against prompt-lookup style drafting. The methods it does cover differ in what information the draft component receives from the target model, how that information is incorporated into drafting, and whether candidate tokens are generated sequentially, autoregressively, in parallel or through a hybrid approach. The post also shows how to enable the methods tested and discusses practical tuning and observability, so the operational path is documented even where the numbers do not generalize.

The experiments run on MI300X and MI355X GPUs under ROCm, and the post presents them as measurements from its test environment rather than as a portable multiplier. Acceptance behavior sits alongside model family, draft checkpoint and workload in the list of factors the outcome depends on, so a result obtained for one target model does not carry to the next without re-measuring. The per-method throughput figures are not reproduced in the material available for this note, so no ranking of native MTP against EAGLE-3 or DFlash is quoted here.

The post groups the drafting methods into three categories. Native MTP modules are built directly into the target-model architecture and use a model-native auxiliary prediction path. Separate MTP drafters use a distinct checkpoint paired with a specific target model, drawing on target-model activations and shared KV-cache information during inference. Dedicated target-conditioned draft networks, namely EAGLE-3, DFlash and DSpark, are separate speculator models trained for a specific target model, and the categories describe the draft component rather than the target family: one target model may support native MTP while also having separately trained EAGLE-3, DFlash or DSpark drafters.

Standard autoregressive decoding commits one token per decode step: four output tokens require four sequential passes through the target model. Speculative decoding separates proposal from verification. A lightweight draft component proposes candidates, the target model checks them in one verification pass from left to right, and accepted tokens are committed. In the post's example, the prompt "The weather today is" gets the draft proposal "sunny and warm outside"; the first two tokens are accepted, the target model selects "clear" at the third position instead of "warm", and "outside" is discarded because it follows the first rejected position.

Native MTP is available in vLLM when the target model includes a compatible auxiliary prediction component. At the first speculative step, the MTP component combines a hidden representation from the target model with information from the current token; at subsequent steps, the newly drafted token and the hidden state produced by the previous MTP step are used to predict the next candidate. In many implementations, parts of the MTP path share components with the target model, which can keep the additional memory overhead relatively modest. Generating multiple speculative tokens still requires sequential drafting before the single verification pass.

Gemma 4 uses a separately packaged MTP draft component paired with a specific target model. It has its own checkpoint but consumes activations produced by the target model and shares the target model's KV cache, which lets it reuse contextual information the target has already computed instead of processing the accepted prefix independently. As with native MTP, the number of layers in the draft component is separate from the configured speculative length, and when several candidates are requested they are generated sequentially before verification.

EAGLE-3 uses a dedicated draft network trained for a specific target model. During the target-model forward pass it records hidden states from three stages of the target Transformer, near the beginning, around the middle and near the end, then concatenates and projects them into a single fused target feature. That fused feature is combined with the embedding of the sampled token before entering the EAGLE-3 draft decoder: the feature summarizes the accepted sequence across several stages of target processing, and the embedding identifies the token drafting continues from.

Draft tokens are then produced autoregressively. Because the target model has not yet processed the later speculative positions, target-model hidden states are unavailable there, so EAGLE-3 feeds the previous draft-component output into the next drafting stage and later draft tokens depend directly on earlier drafted ones. DFlash instead drafts parallel blocks from target-model hidden states, and DSpark adds lightweight causal correction and confidence-based prefix selection. The source material available here stops before showing how those architectural differences played out in throughput on MI300X and MI355X.

The post is explicit that the number of physical MTP layers and the configured speculative length are separate concepts. When num_speculative_tokens exceeds the prediction depth the checkpoint provides directly, vLLM can reuse the MTP path through additional forward passes. A larger value proposes more candidates before verification but introduces more sequential drafting work, and every candidate past the first rejected position is discarded. That is the knob most directly coupled to the serving loop, and the post gives no recommended value, so it has to be swept per target model and draft checkpoint pair.

The loudest thread of complaint is about hardware coverage rather than drafting methods. Commenters on Hacker News argue that the workstation-grade AMD R9700 AI Pro has been ignored by both the vLLM and AMD teams, with one saying stock vLLM runs slowly on those cards compared with forks such as Radiance and reporting a jump from roughly 20-30 tokens per second of generation to 150-200. Another commenter in the same thread warns about massaged vendor numbers, citing a claim they attribute to Intel of about 1400 tokens per second of generation for Qwen3.6-moe on Arc b70 that, on inspection, turned out to be 200 simultaneous users running the same 1024-token prompt. Neither figure appears in the vLLM post, which reports MI300X and MI355X only.

The other recurring question is how verification can be cheap if the target model still has to run. Commenters answer it with the mechanism the post implies: the context and all candidate tokens pass through the backbone in parallel in a single forward pass, which amortizes the cost of loading weights from VRAM across those tokens before the language head produces the distributions used to accept or drop them. One commenter draws the operational conclusion directly, saying that if you already have another source of parallel data, such as many concurrent users or separate tasks, speculative decoding might not provide any benefit. The vLLM post, in the material available here, publishes no batch-size breakdown that confirms or refutes that, and it is the gap most relevant to production ROCm serving.

The post names the hardware, the ROCm platform and the five drafting methods, and it commits to the claim that results depend on drafting method, proposal length, model family, draft checkpoint, workload and acceptance behavior. It does not, in the material available here, provide a per-method throughput ranking that can be carried into a capacity model, a batch-size regime at which drafting stops paying, or any data on non-datacenter AMD parts such as the R9700 that commenters raised.

For teams standing this up on ROCm, the tractable plan follows from what the post does say. Instrument acceptance length per request alongside output-token throughput, since acceptance is the variable the post singles out and the one that determines how many committed tokens each verification pass yields. Sweep num_speculative_tokens for the specific target and draft checkpoint pair rather than reusing a value from another model family, and re-run the sweep at the concurrency actually served rather than at batch size one.