Embedding LLMs and RAG directly into Apache Beam transforms for low-latency inference on high-velocity data streams.
The problem
The usual shape for streaming AI is a pipeline that leaves the stream: batch the events, call an external inference service, wait, come back. Every hop adds latency and a new failure mode.
How it works
- LLM inference is embedded directly inside Apache Beam transforms, so reasoning happens in the stream rather than in a round trip out of it.
- Retrieval runs inline too — the pipeline carries its own RAG stage instead of calling a separate service per element.
- Model handles are shared across bundles so a transform does not pay initialisation cost per element, which is what usually makes in-pipeline inference impractical.
- The result is low-latency intelligence on high-velocity data without the data ever leaving the pipeline.
Why it exists
The demo and slides from my Apache Beam Summit 2026 technical session in New York.
More work
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.