Lightning talk · 2026
Scaling Production RAG Systems with Kubernetes
How retrieval-augmented generation earns its uptime: autoscaling vLLM, vector search, and observability on Kubernetes so a RAG system stays fast and affordable when traffic stops being a benchmark and starts being real.
How retrieval-augmented generation earns its uptime: autoscaling vLLM, vector search, and observability on Kubernetes so a RAG system stays fast and affordable when traffic stops being a benchmark and starts being real.
What the talk covers
- vLLM served behind FastAPI on Kubernetes, using continuous batching and KV-cache reuse so GPU time is not wasted on padding.
- Horizontal autoscaling driven by real serving signals — queue depth and latency — rather than CPU utilisation, which tells you almost nothing about an inference workload.
- Vector search and reranking deployed as their own scalable units, so retrieval and generation can be sized independently.
- Tracing and evaluation wired in from the start, so a quality regression surfaces as an alert rather than a support ticket.
Why it matters
A RAG demo is easy. A RAG system that stays fast and affordable when traffic stops being a benchmark and starts being real is an infrastructure problem, and it is where most teams stall.
The other talk
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.