GLM-5.3 KV cache exposure leaked 155 Workers AI responses via stale cache pages

The exposure ran from the model's launch on August 27, 2026 until September 9. Gadi Evron of Knostic reported it through Cloudflare's HackerOne program on September 8 at 23:11 UTC. Cloudflare declared an incident at 13:22 UTC on September 9, set the model to private at 14:00, and restored it at 19:40 with a corrected configuration and a patched engine. Cloudflare says no other Workers AI model was affected. After excluding its own and the researcher's testing, it says the affected requests are consistent with ordinary long-context use, and it found no evidence of deliberate exploitation.

Workers AI serves GLM-5.3 (@cf/zai-org/glm-5.3) with SGLang in a disaggregated setup. Prefill GPUs build the KV cache for a prompt and hand it over RDMA, via Mooncake, to decode GPUs that generate the response. SGLang sizes the KV pool at startup from whatever GPU memory remains after loading weights, while the context length is a separate configuration value. Nothing forces the two to agree. On the GLM-5.3 prefill nodes they did not: 681,280 tokens of pool against 1,048,576 tokens of context.

One commenter on Hacker News asked whether Cloudflare advertised a 1,048,576-token window it could not hold. The report says the model was deployed with a 1,048,576-token context window and that Cloudflare knew about the mismatch before launch. The team expected the engine to return an error for prompts it could not process. Pre-launch tests showed successful HTTP status codes and a streaming body with no error frames. Those tests checked status codes and performance at long contexts, not model output and coherence, and Cloudflare now checks pool size against context length from the running engine before any disaggregated deployment change ships.

When a prompt exceeded the pool, SGLang's scheduler rejected it through Req.set_finish_with_abort. That call recorded the error and swapped the prompt for a one-token placeholder so the request could drain through normal batching. On a single node this produced an error for the client. In disaggregated mode, both admission queues checked whether a request was finished, not whether it was rejected. The rejection lived in to_finish rather than finished_reason, so finished() returned False and the request looked live.

Cloudflare says every request over the threshold behaved this way, so the handoff happened every time rather than by a timing accident. One visible signal was that the reply arrived faster than an 8-page prefill should take. Another user had reported the bug to SGLang as issue #30233 on July 6, 2026, which Cloudflare found while investigating.

The KV cache is a shared pool of fixed-size pages. Pages are assigned at admission, released at completion, and not zeroed between requests; the next owner simply overwrites them. Cloudflare describes this as common among inference engines and necessary for the best performance. That makes the engine's page ownership and lifetime tracking the isolation mechanism. The tracking has to handle tasks on the same node with different lifetimes, such as KV cache offload and token generation, and a remote prefill node that holds a simultaneous mutable reference to the page.

One commenter on Hacker News concluded that enterprises with highly sensitive data should use non-shared hosts. The report does not address dedicated hosting or offer a page-zeroing option. Other commenters suggested that unrelated responses they had received from other providers might have the same cause. The sources say nothing about any provider besides Cloudflare, so treat that as speculation.

Cloudflare calls pool size alone a fragile guarantee. Changes to quantization, tensor parallelism, CUDA graph sizes or a future model version all change the pool size in ways that are easy to miss. Other situations may also exist in which the prefill scheduler rejects a request and triggers the same bug. Every other disaggregated Workers AI deployment already had pools larger than its context length on both prefill and decode. The engine fix and later hardening now apply to every model Cloudflare serves with SGLang, and its deployment checks require them.

Cloudflare says the patch was released in SGLang v0.5.20, with follow-ups in v0.5.21 (October 2, 2026) and v0.5.22 (October 19, 2026). It also says the bug was addressed in v0.5.21. Between September 16 and October 12, upstream merged the admission fix, per-transfer checksums and abort-ACK preservation, and turned deferred KV release on by default. Cloudflare met the SGLang maintainers, including RadixArk's CTO and core engineers, at 23:30 UTC on September 10. Teams serving shared-cache inference with SGLang in disaggregated mode should upgrade to v0.5.22 or later and compare the KV pool size against the configured context length; SGLang prints the pool size in its startup logs and exposes it as a metric.

Cloudflare groups this incident with KV cache memory-safety bugs it has been fixing for months. In each, a page is freed, reused or read while something else still claims it. One cause is lifetime within a node, where generation, host offload, the transfer receiver and abort paths share pages. Earlier fixes include #24226 and #30899 for decode offload and #30937 for double release on grammar errors. The other cause is a cross-node race: an abort arrives while prefill is still writing over RDMA, and late writes land in a page already given to another request. Cloudflare proposed #31357 and #38961 and contributed a reproduction script in #32564.

In all of these the result is corrupted output or one user's data in another user's response, with no error raised. Tests that check only status codes will pass. Long-context validation therefore has to check output content, as Cloudflare's canary-marker tests did before the model was restored.