the short version
- The default max_num_batched_tokens is now 16384, double the previous 8192, which changes activation memory and scheduling behaviour without any config edit on your part.
- bitsandbytes quantization is no longer in-tree; it now requires an out-of-tree plugin, so any bnb-loaded deployment breaks on upgrade.
- calculate_kv_scales and override_attention_dtype were removed outright; a config file containing either will need editing before v0.28.0 runs.
- Transformers is pinned to 5.15.0, which makes this an environment-wide upgrade rather than a drop-in vLLM bump.
Published
Key facts
- vLLM v0.28.0 shipped 26 August 2026 with 584 commits from 270 contributors
- max_num_batched_tokens default doubled from 8192 to 16384 (#51726)
- bitsandbytes moved to an out-of-tree plugin (#43529) and Transformers pinned to 5.15.0 (#51668)
- Kimi-K3 work reports 1.5 to 3x kernel-level all-gather speedup (#51070) and ~60% better DSpark TTFT (#51725)
- Optional shared-expert sharding saves roughly 17 GiB per GPU (#50912)
vLLM v0.28.0 shipped on 26 August 2026 with 584 commits from 270 contributors. The change most operators will feel immediately is a default: max_num_batched_tokens went from 8192 to 16384 (#51726). Alongside it, prefix caching is now on by default for Mamba models (#50991) and the Blackwell CUDA graph capture default was raised to 1024 (#49390).
Four things were removed or moved. bitsandbytes support migrated to an out-of-tree plugin (#43529). Transformers was bumped to 5.15.0 (#51668). The deprecated calculate_kv_scales runtime KV scale calculation was removed (#49389), as was override_attention_dtype (#48684). Each of these is a config or environment edit that has to happen before the upgrade, not after.
What the new defaults change at runtime
max_num_batched_tokens is the per-step token budget the scheduler fills. Doubling it to 16384 means the engine will pack more tokens into a single forward pass by default, which raises peak activation memory and shifts the balance between prefill throughput and per-request decode latency. Anyone who previously tuned this value explicitly is unaffected - the flag still wins. Anyone who relied on the 8192 default and sized their GPU memory headroom around it should set the value explicitly rather than inherit the new one.
Prefix caching being enabled by default for Mamba models is the second default worth reading carefully. It is a behaviour change for a specific model family, not a global one, and it means Mamba deployments that previously ran without prefix cache now allocate for one. The release notes state the change but do not report a measured throughput delta, so the effect on your workload is not something the source quantifies.
The Blackwell CUDA graph capture default rising to 1024 affects startup time and captured-graph memory on Blackwell hardware. Again, the release lists the new value and nothing about the tradeoff it produces, so treat capture time on first boot as something to measure rather than assume.
The four breaking changes that need config edits first
- bitsandbytes is out-of-tree (#43529). If you load bnb-quantized weights, the quantization path is no longer part of the vLLM package and requires the plugin.
- Transformers is now 5.15.0 (#51668). This is a pinned dependency bump, so shared environments and any code depending on older Transformers APIs need checking before the vLLM upgrade.
- calculate_kv_scales was removed (#49389). Runtime KV scale calculation is gone; configs still passing it will need the flag stripped.
- override_attention_dtype was removed (#48684). Any deployment forcing an attention dtype through this option must find another route.
The pattern here is that three of the four are silent in the sense that they only surface at engine start. Test the upgrade against a config that mirrors production, not a minimal one, because the failure mode for a removed flag is a start-up error rather than a degraded response.
Memory and throughput work aimed at specific models
The largest single engineering effort in the release is a Kimi-K3 performance push spanning the stack. It includes Decode Context Parallel (DCP) support (#50484), fused FlashKDA decode and prefill kernels (#50654, #51311, #52458), SiTU activation support for MegaMoE (#50510), and GEMM-RS for sequence parallelism (#52079). Two items carry numbers: combined all-gathers with a 1.5 to 3x kernel-level speedup (#51070), and an adaptive speculative token budget delivering roughly 60% better DSpark TTFT (#51725).
The memory item to note is optional shared-expert sharding, which saves approximately 17 GiB per GPU (#50912). It is optional, which means it is a config decision rather than a default - worth evaluating if Kimi-K3 memory pressure is what is currently forcing a larger tensor-parallel degree than you want. Kimi-K3 also now runs on ROCm with the V2 model runner (#51653).
DeepSeek V4 gets sparse MLA working end-to-end for plain decode, MTP, and DSpark speculative decoding (#51538), plus AMD Quark NVFP4 support (#47972), sparse top-k metadata kernel optimizations (#52084, #51967), and narrowed eager CUDA graph regions (#51430, #52401). ROCm enablement extends to gfx11 and gfx950 (#47017, #52212).
Where the memory hierarchy is heading
Tiered KV cache offloading picked up disk offloading support (#49644), out-of-tree secondary tier managers via module_path (#51007), partial secondary-tier load results (#50321), tiering metrics (#48798), and a canonical CPU layout for parallelism-agnostic offload (#48414). The module_path hook is the interesting one for anyone with a custom storage tier - it is an extension point rather than a fixed implementation, and the canonical CPU layout means offloaded blocks are no longer tied to the parallelism configuration that produced them.
Model Runner V2 continues to fill out: E/P/D disaggregation (#38390), weight offloading (#51413), multi-layer MTP KV cache support (#50062), encoder CUDA graphs (#49852), decoder token-wise pooling (#50931) with Transformers pooling models (#52425), attention-free models (#52374), and thinking_token_budget support (#46727). Speculative decoding gains DFlash2 with local convolution and a candidate selector (#52816), DSpark confidence-scheduled verification (#47808), and async scheduling auto-enabled for draft models (#48341) - that last one is another default change, applying specifically to draft models.
What the release notes do not tell you
The notes give commit numbers and feature names but almost no end-to-end benchmarks. There is no reported tokens-per-second delta for the max_num_batched_tokens change, no latency figure for prefix caching on Mamba, and no guidance on what hardware the 16384 default assumes is available. The two numbers that are stated - the 1.5 to 3x all-gather speedup and the ~60% DSpark TTFT improvement - are scoped to Kimi-K3 and are kernel-level and TTFT-level respectively, not full-system throughput.
Practical sequencing for an upgrade: pin max_num_batched_tokens explicitly so the default change is neutral, grep configs for calculate_kv_scales and override_attention_dtype, confirm whether anything in the serving path loads bitsandbytes weights, and check Transformers 5.15.0 against the rest of the environment. Distribution is available as CUDA 13.0 and CUDA 12.9 wheels, ROCm wheels from wheels.vllm.ai, and Docker images for CUDA 13.0 (default), CUDA 12.9, Ubuntu 24.04 variants, ROCm, CPU, and XPU.
Questions this raises
what changed in vllm v0.28.0
The default max_num_batched_tokens doubled from 8192 to 16384, prefix caching is now on by default for Mamba models, and the Blackwell CUDA graph capture default rose to 1024. Four items were removed or moved: bitsandbytes went out-of-tree, Transformers was bumped to 5.15.0, and both calculate_kv_scales and override_attention_dtype were removed.
does max_num_batched_tokens 16384 break my config
No, if you set the value explicitly the flag still wins and you are unaffected. If you relied on the 8192 default and sized GPU memory headroom around it, pin the value explicitly before upgrading, because the larger budget packs more tokens per forward pass and raises peak activation memory.
how do I use bitsandbytes with vLLM 0.28.0
bitsandbytes support migrated to an out-of-tree plugin in #43529, so the quantization path is no longer part of the vLLM package. Loading bnb-quantized weights now requires installing that plugin, and the failure surfaces at engine start rather than as a degraded response.
These daily notes are drafted by a model I run and operate myself - the same kind of pipeline this site is about - from sources published in the previous 24 hours, and every one lists what it read. The longer essays, the talks and the preprint are mine, written by hand.