the short version
- Ollaya serves /v1/systemone and /v1/models in TypeSafe's request and response shapes, and the TypeSafe Python SDK 0.7.1 works unchanged against a local server.
- The 8 to 10 ms figure belongs to laya, a 322m or 421m model run in fp16 on an RTX 4090. The hosted Jev figure it is compared against, 236 to 276 ms, includes network time.
- The sources contain no benchmark of these models served through Ollama or vLLM, only commenter claims about possible future support.
- A leaderboard posted by one commenter puts Laya 421M at 30.25 against Jev 1.13 at 63.29, and the developer agrees Laya is a lot weaker than Jev on harder queries.
Ollaya is a local server with an Apache-2.0 runtime that runs open-weight, Jev-style decision models on ONNX Runtime, on the CPU or an NVIDIA GPU. It exposes TypeSafe's /v1/systemone endpoint, so the TypeSafe Python SDK 0.7.1 works unchanged against it. On an RTX 4090, laya answers a five-question request in 8 to 10 ms through the HTTP API, against a 236 to 276 ms median for TypeSafe's hosted Jev API in third-party benchmarks. The developer said on Hacker News that the small models it supports today are well below Jev on harder queries. The sources contain no benchmark against Ollama or vLLM.
A decision model answers typed questions (choice, score, yes/no) about a piece of text or JSON in a single forward pass, with no token-by-token generation. It returns calibrated probabilities, not generated text. The example response on the project's homepage, published 25 September 2026, reports output_tokens as 0. Whether a local server can replace the hosted API therefore depends on two separate things: the latency claim and the accuracy gap that Hacker News commenters raised.
Which open-weight models Ollaya runs
The homepage lists eight model families and links to a models page that shows each one's accuracy and speed. Every listed family is credited to an outside author, from Convai Innovations to the Qwen team. The homepage recommends laya for speed, decider for accuracy, von for long inputs of up to 8,192 tokens, and qwen3guard for safety screening.
- laya (Convai Innovations): 322m and 421m. It gives typed, calibrated answers in English and 100+ languages and is the fastest option.
- decider (Mapika, on Qwen3.5): listed as 0.75b and 1.9b, with latency tags decider:0.8b and decider:2b. It reads the answer from option-letter logits, and decider:2b scores 0.591 on typed decisions.
- nli (Moritz Laurer): 396m and 435m zero-shot entailment classifiers, which the project describes as the most accurate encoder model on typed decisions in its tests.
- gliclass (Knowledgator): 439m. It scores all options of a question in one pass, so cost barely grows with option count.
- qwen3guard (Qwen team): 0.6b safety guard. It classifies text as safe, controversial or unsafe, and names the unsafe category, in 119 languages.
- decision (vLLM Semantic Router contributors): 0.75b, a fully fine-tuned Qwen3.5 backbone with an endpoint head and 16k-token rows.
- kev (Jared Palmer): 0.76b, 4.2b and 7.9b. It is a LoRA on a Qwen3.5 base plus a pointer head, calibrated with Kev's own temperature.
- von (Victor Hugo Panisa, on ModernBERT-large): 395m with an 8k-token context and an input-conditioned calibration.
Weights are pulled from each author's Hugging Face repository, pinned to a commit and checked against sha256. Ollaya never re-hosts them. Models execute on ONNX Runtime, plus MLX for two models on Apple GPUs, but the homepage does not name the weight file format or say whether weights are converted on pull. Each model ships its own calibration, and a Modelfile can refit that calibration on your labelled data.
For teams with compliance review, the Apache-2.0 runtime licence and the licences on the individual weights are separate questions. The homepage does not list the weight licences, so each author's Hugging Face repository has to be checked.
The API surface is TypeSafe's /v1/systemone
The server listens on 127.0.0.1 by default, and the documented examples use port 11435. It serves /v1/systemone and /v1/models with TypeSafe's request and response shapes. To point an existing TypeSafe SDK client at it, you set environment variables, and any API key value works. The documented example is below.
export TYPESAFE_BASE_URL=http://localhost:11435
export TYPESAFE_API_KEY=local # any value works
export TYPESAFE_DEFAULT_MODEL=laya
curl http://localhost:11435/v1/systemone -d '{
"model": "laya",
"state": "Can I get an invoice for last month?",
"questions": {
"intent": {
"type": "choice",
"instructions": "What does the customer want?",
"criteria": {
"invoice": "Needs an invoice or receipt",
"refund": "Wants money back",
"other": "Anything else"
}
}
}
}'The response resolves the model alias to laya:en. It returns choice invoice with confidence 0.9547 and per-option probabilities of invoice 0.9698, refund 0.0172 and other 0.013. It also includes a usage block with 43 input tokens and 0 output tokens. Because the shape matches the hosted API, a client can hold both endpoints and route between them, but the sources do not address whether probabilities from the two backends are comparable.
Hardware needs: CPU everywhere, NVIDIA via CUDA 13
- Every model runs on the CPU, on macOS (Apple silicon, macOS 14+), Windows 10 and 11 x64, Linux x86-64, Linux ARM64, WSL 2 and Docker (amd64 and arm64).
- NVIDIA acceleration uses CUDA 13 and needs driver R580 or newer. It works on Linux, Windows, WSL 2 or the Docker :cuda image, which is amd64 only. The install scripts fetch the CUDA libraries only when they find an NVIDIA GPU.
- On a Mac, laya and nli run on the Apple GPU through MLX, and other models use the CPU.
- AMD and Intel GPUs, Linux ARM64, and the Windows and Linux desktop apps use the CPU.
The published medians for a five-question request on an RTX 4090 are laya:multilingual 8.1 ms, laya:en 9.6 ms, gliclass 14.7 ms, nli 20.4 ms, decider:0.8b 155 ms and decider:2b 190 ms. Laya ran in fp16 and the others in fp32. The homepage's terminal example also shows decider:2b answering in 178 ms. The Jev figure comes from third-party benchmarks of the hosted API (AbdelStark/jev-benchmarks and nibzard/decision-model-benchmark) that include the network, and the page calls it an order-of-magnitude comparison.
The sources give no memory or VRAM figure for any model, and no CPU-only latency. Listed sizes run from laya's 322m to kev's 7.9b, so sizing hardware for the largest models is guesswork. One commenter said the open models that come close to Jev are big models, which makes the missing memory figures matter more.
How Ollaya compares with Ollama and vLLM
The sources contain no benchmark against Ollama or vLLM and do not show either serving these models. Commenters on Hacker News argued that Ollama could add decision-model support at any time. One reply said Ollaya uses the same API as Jev, so client code is not tied to it either way.
On vLLM, one commenter wrote that the next vLLM release will have this if you use gateways. The same commenter said the GoModel gateway supports the S1 endpoints, with virtual models that keep a stable name while the backing model is swapped. Separately, the decision family in Ollaya's catalogue comes from vLLM Semantic Router contributors. From the sources, the portable layer is the TypeSafe request shape, not any one server.
Practitioners question Laya's accuracy against Jev 1.13
The strongest pushback was on quality. One commenter posted a jevbench leaderboard with decider-4b v2 at 64.13, Jev 1.13 at 63.29, Kev 4B at 36.14 and Laya 421M at 30.25. decider-4b is not among the decider sizes listed on the homepage. Another commenter built a semantic grep tool on both backends and found Laya fell apart on looser queries such as "mans name". A third said Laya did not come close for their use case but decider was just as good.
A commenter identifying as the developer agreed that Laya is a lot weaker than Jev, especially on harder queries. They described it as a small, fast model and said running the bigger open models that get close to Jev is what they are working on next.
The small ones I support today are well below Jev on harder queries, but fine for simple, well-defined questions.
Commenters raised four further points. One suggested that if you have an eval set, training a classifier is the smarter move, and a reply agreed for fixed tasks. Another called "decision model" marketing jargon for a classifier. A third said the homepage example was really text classification, and a reply said it would be changed. A fourth argued that fine-tuning laya on LLM-generated data is cheap and quick, though no numbers were given.
Still unknown: memory, larger models, independent evals
Several questions remain open in the sources:
- Memory and VRAM requirements for each model, including kev at 7.9b.
- CPU-only latency for any model.
- The weight file format and the licence on each model's weights.
- How the larger open models, which the developer says are coming next, perform on 4090-class hardware.
- Whether Ollama adds native support, and what the next vLLM release actually ships.
- Whether the jevbench leaderboard figures hold up under independent evaluation.
Until those are answered, evaluate each model on your own labelled questions and use the Modelfile calibration refit before setting thresholds. Keep the TypeSafe client shape so the backend can be swapped between Ollaya, the hosted API or a gateway.
Questions this raises
Is Ollaya as accurate as Jev?
Not for the small models. A commenter identifying as the developer said on Hacker News that Laya is a lot weaker than Jev, especially on harder queries. A posted jevbench leaderboard showed Laya 421M at 30.25 against Jev 1.13 at 63.29.
What hardware does Ollaya need?
Every model runs on the CPU on macOS 14+ with Apple silicon, Windows 10 and 11 x64, Linux, WSL 2 and Docker. NVIDIA acceleration needs CUDA 13 and driver R580 or newer. The sources give no memory, VRAM or CPU-only latency figures.
Does Ollaya work with the TypeSafe SDK?
Yes. Ollaya serves /v1/systemone and /v1/models with TypeSafe's request and response shapes, so TypeSafe Python SDK 0.7.1 works unchanged after setting environment variables. Any API key value is accepted.
These daily notes are drafted by a model I run and operate myself - the same kind of pipeline this site is about - from sources published in the previous 24 hours, and every one lists what it read. The longer essays, the talks and the preprint are mine, written by hand.