Demo from my KCD New York 2026 talk — deploying, autoscaling, and observing production RAG pipelines on Kubernetes with vLLM and vector search.
The problem
A RAG demo is easy. A RAG system that stays fast and affordable when traffic stops being a benchmark and starts being real is an infrastructure problem, and it is where most teams stall.
How it works
- vLLM served behind FastAPI on Kubernetes, using continuous batching and KV-cache reuse so GPU time is not wasted on padding.
- Horizontal autoscaling driven by real serving signals — queue depth and latency — rather than CPU utilisation, which tells you almost nothing about an inference workload.
- Vector search and reranking deployed as their own scalable units, so retrieval and generation can be sized independently.
- Tracing and evaluation wired in from the start, so a quality regression surfaces as an alert rather than a support ticket.
Why it exists
The demo repository from my KCD New York 2026 lightning talk — the full deployment, not the slides.
More work
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.