Production RAG on Kubernetes: vLLM, Scaling, Observability

Retrieval-augmented generation is easy to demo and expensive to run. The demo is a notebook: embed a corpus, stand up a vector index, call a model. Production is a different problem - the moment traffic stops being a benchmark, you are operating a GPU fleet with bursty, long-tail request costs, a retrieval layer whose quality silently decays, and a latency budget that users experience one token at a time. This is a reference for that second problem, expanded from my KCD New York 2026 talk. It assumes you know what RAG is and want it to survive real load on Kubernetes.

A scoping note on sources: the architecture and operational guidance here is what I presented on stage and what the components document about themselves - vLLM flag and metric names are from the vLLM docs, KEDA behavior from the KEDA docs. Where a number depends on your model, hardware and corpus, I say so instead of inventing one. Verify flag names against the vLLM release you deploy; projects rename things.

What does a production RAG deployment look like

Four planes, scaled independently. Collapsing them into one deployment is the most common architectural mistake, because their scaling axes have nothing in common.

The gateway and retrieval planes are ordinary stateless services - a plain Horizontal Pod Autoscaler on CPU works. Everything unusual about RAG operations concentrates in the last two planes: the vector store because it is stateful and memory-bound, and the vLLM pool because its real capacity metric is invisible to the default autoscaler.

Why CPU autoscaling fails for vLLM

An LLM inference pod under heavy load shows modest CPU: the work happens on the GPU, and the thing that actually runs out is KV-cache memory and batch slots. An HPA watching CPU will conclude the pod is idle while requests queue. You need to scale on the signal vLLM itself exposes - its Prometheus endpoint reports, among others, vllm:num_requests_running, vllm:num_requests_waiting, and time-to-first-token histograms. The waiting-queue depth is the cleanest scale signal: when requests wait, you need another replica; when the queue is empty across the pool, you can shrink.

KEDA is the practical way to wire that up, because it speaks Prometheus natively and handles scale-from-and-to-low-counts better than a raw custom-metrics HPA pipeline. The shape:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-pool
spec:
  scaleTargetRef:
    name: vllm
  minReplicaCount: 1        # scale-to-zero costs a cold model load
  maxReplicaCount: 8        # your GPU budget, stated explicitly
  cooldownPeriod: 300       # longer than a typical burst, shorter than a lull
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting)
        threshold: "4"      # queued requests per replica you will tolerate

Two operational caveats that the clean YAML hides. First, replica startup is not pod startup: a vLLM replica is ready when the model weights are loaded into GPU memory, which for a 7B-class model is tens of seconds from a warm image with weights on fast local storage, and much worse if the pod pulls weights over the network on every start. Bake weights into the node (hostPath cache, read-only PVC, or an image layer) and gate readiness on the health endpoint, not on the container starting. Second, scale-to-zero sounds like free money and usually is not: the first request after an idle period pays the whole cold-load, so keep minReplicaCount at 1 unless the workload is genuinely batch-shaped and the users are machines that can wait.

Where should the vector store live

Three honest options, and corpus size picks between them more than ideology does.

The memory arithmetic is worth doing before choosing: stored vectors cost dimensions x 4 bytes each for float32 before index overhead - a million 768-dimensional embeddings is roughly 3 GB of raw vectors, and graph indexes like HNSW add a meaningful multiple on top. If that number times your replica count is uncomfortable, the per-pod option is out regardless of preference.

Whatever the store, keep the embedding model pinned and versioned alongside the index. An index built with one embedding model and queried with another does not fail loudly - it just returns quietly worse neighbors, which is the worst failure shape in this whole system.

The four numbers that tell you RAG is healthy

• Time to first token (TTFT), p50 and p99 - the user’s perception of speed in a streaming UI. vLLM exposes it directly as a histogram.
• End-to-end latency split by stage - embed, search, rerank, generate - because "RAG is slow" is not actionable and "rerank added 400ms" is. This needs a trace span per stage, not a single timer.
• Token throughput per replica (generation tokens per second) - your capacity and cost denominator, from vLLM’s generation-token counters.
• Retrieval quality - hit-rate against a labeled set, or at minimum groundedness sampling of production answers. This is the number that decays silently as the corpus and the world drift apart; the infrastructure metrics all stay green while answers get worse.

The first three come free from instrumentation. The fourth is the one teams skip because it needs an evaluation set, and it is also the one that decides whether the system is any good. A small, honest labeled set - even a few hundred query-document pairs curated once a quarter - beats a dashboard of infrastructure metrics for answering "did last week’s reindex make retrieval worse".

What a request actually costs

The GPU nodes dominate; everything else is noise by comparison. So the cost model reduces to utilization: cost per 1,000 generated tokens = (GPU node hourly price / 3600) x 1000 / sustained tokens-per-second per node. The two levers that move it are batching efficiency (vLLM’s continuous batching does this work for you - protect it by not capping max-num-seqs lower than your concurrency needs) and right-sizing the model to the task. A smaller model that answers well at three times the throughput is a two-thirds price cut no procurement negotiation will ever match; this is where an evaluation set pays for itself twice.

Failure modes worth rehearsing

• Long-context burst -> KV-cache exhaustion: a handful of maximum-length requests can preempt or stall a batch. Cap max-model-len and max-num-batched-tokens to what your product actually needs, not what the model supports.
• Deploy-time thundering herd: rolling a vLLM pool restarts model loads; with default surge settings you can briefly halve capacity under full load. Roll one replica at a time, gate on readiness, and deploy off-peak.
• Retrieval staleness: the corpus updates, the index does not, and nobody notices for a month - because no infrastructure metric covers it. Alert on index age, not just index availability.
• Silent embedding-model drift: see above - version the embedding model with the index, and treat a mismatch as a deploy blocker.
• Vector-store restarts on stateful nodes: an HNSW index that rebuilds or reloads on restart takes real minutes at scale; know your store’s recovery time before an incident teaches it to you.

The talk and the demo, honestly labeled

The recording below is the KCD New York 2026 lightning talk this reference grew from. Its demo repo is deliberately a teaching skeleton - FastAPI, sentence-transformers and FAISS in a single service - which is the right shape for showing the request path on stage and explicitly not the four-plane production shape described above. Use the repo to see the moving parts in one file; use this page when you split them apart for real load.

► PLAY RECORDING