On-device inference with zero cloud dependency. Runs LLaMA 3.1, IBM Granite, and Mistral via 4-bit GGUF at ~8 tok/s, with on-device RAG over your own documents.
The problem
Every mainstream phone assistant is a thin client for someone else’s datacentre. Your prompt leaves the device, and the moment the network does too, the assistant stops existing. PrometheusAI asks whether a useful assistant can run entirely on the handset in your pocket.
How it works
- 4-bit GGUF quantisation brings LLaMA 3.1, IBM Granite, and Mistral down to a footprint a phone can actually hold in memory, at roughly 8 tokens per second on-device.
- A native Kotlin inference layer runs the model locally — no API key, no request leaving the handset, and no degradation when the network does.
- On-device retrieval-augmented generation indexes the user’s own documents locally, so the model can answer from personal context that would never be acceptable to upload.
- Model swapping is first-class: the same runtime hosts several open-weight models so the trade-off between size, speed, and quality stays the user’s to make.
Why it exists
It is the clearest expression of the argument behind my research: the interesting frontier in applied AI is not a larger model in a larger datacentre, it is capable inference on hardware people already own.
More work
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.