Technical session · 2026
Real-Time AI Pipelines at Scale: Embedding LLMs into Apache Beam
Bringing LLM inference and RAG inside the pipeline itself - embedding models directly into Apache Beam transforms so streaming data can reason in real time, delivering low-latency intelligence on high-velocity data without ever leaving the stream.
What the talk covers
- LLM inference is embedded directly inside Apache Beam transforms, so reasoning happens in the stream rather than in a round trip out of it.
- Retrieval runs inline too - the pipeline carries its own RAG stage instead of calling a separate service per element.
- Model handles are shared across bundles so a transform does not pay initialisation cost per element, which is what usually makes in-pipeline inference impractical.
- The result is low-latency intelligence on high-velocity data without the data ever leaving the pipeline.
Why it matters
The usual shape for streaming AI is a pipeline that leaves the stream: batch the events, call an external inference service, wait, come back. Every hop adds latency and a new failure mode.
The other talk
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.