topic · 19 notes
Local Inference, as it ships.
Notes on running models on hardware you own - Apple Silicon with MLX and Metal, llama.cpp and GGUF builds, Ollama and other local runtimes - and on the quantization and memory trade-offs that decide what fits on a laptop or a single workstation GPU.
Strands Decider 2B makes local decisions in a median of about 115 ms on an RTX 3090
Strands Decider 2B is an open 2B decision model: 115 ms median on an RTX 3090, 3rd of 33 on JevBench, but no named licence or p99 latency in the release post.
The ds4 LLM engine generates 39.4 t/s at q2 on a 128 GB M5 Max
ds4 LLM engine (DwarfStar 4) generates 39.4 t/s and prefills 790.2 t/s at q2 on a 128 GB M5 Max. MIT-licensed C for Metal, CUDA and ROCm, short model list.
What the Magnitude inference engine tunes, and what to check before production
Magnitude inference engine tunes kernels on-device in about a minute and claims up to 2x llama.cpp, but no prompt, routing or harness rewriting is described.
What the Jeff decision model gives up for 30 ms, and when it can stand in for Jev
Jeff decision model: an open 0.8B Jev-compatible classifier that decides in 22 ms but scores 47.6 vs 73.3 on JevBench hard. When it can replace Jev.
What Ollaya runs locally for Jev-style decision models, and what it needs
Ollaya serves open decision models on TypeSafe's /v1/systemone API. Laya answers five questions in 8 to 10 ms on an RTX 4090 but trails Jev on hard queries.
M5 Ultra Mac Studio local LLM benchmarks: prefill, bandwidth and the 512 GB ceiling
M5 Ultra Mac Studio prompt processing is up 150% over M3 Ultra, generation about 70%, bandwidth 819 GB/s to 1.2 TB/s, memory ceiling still 512 GB.
The Qwen-Image-2.1 license is clear, the weights and VRAM numbers are not
Qwen-Image-2.1 shipped 20 September 2026. Its LICENSE bars commercial use without a separate Qwen license; the 7B figure is a commenter claim, VRAM unknown.
What the CUA-S1 system one computer use model ships, and what it does not
trycua's CUA-S1 is an MIT-licensed, source-only research release; CUA-S1-FORMS weights sit on Hugging Face and the README has no latency or accuracy numbers.
How a 4B model trained with RL produced query plans faster than Postgres
A 4B Qwen model post-trained with SFT and a custom GRPO variant hit a 1.81x geometric mean speedup and 44.7% lower summed latency on 113 join-heavy queries.
What a Linux GPU driver for the M4 Mac Mini gives you today
A Linux GPU driver for the M4 Mac Mini hits OpenGL ES 3.0 conformance and 200fps Minecraft in about a month, but has no compute API and no Vulkan.
Migrating large system prompts to Ollama: 35KB eats 14% of context
A 35KB preprompt eats 14% of Ollama's 65k-token window, agents thrash within 3 minutes. Field notes on context accounting, SOP rewrites and opencode setup.
Deltafin Kimi K3 SSD weight streaming on a MacBook: 1 tok/s, 376s first token
A deltafin fork streams Kimi K3's 1.45 TB expert bank off four SSDs on an M5 Max MacBook Pro at 1.00 tok/s, with 376s to first token on 512 tokens.
Run 104GB Qwen3.8-Flash-Next on a 48GB Mac via slotstream at ~12 tok/s
slotstream keeps a 3.8GB dense trunk resident and streams 68GB of experts from SSD into a 20.1GB slot pool for ~12 tok/s decode on a 48GB M5 Pro.
Qwen 3.8 27B Reverse-Engineered a License Check in 30 Minutes, Locally
Qwen 3.8 27B ran on one Nvidia GB10 workstation, statically reverse-engineered a paid app's license scheme and built a bypass in about 30 minutes.
The local serving defaults that make an open-weight model feel dumber than it is
A Level1Techs post argues sampler settings, chat templates, attention backends and undocumented KLD claims explain why local open-weight models underperform.
GPT-5.6 Sol drops 20% per token, and the agent stacks that won't feel it
OpenAI cut GPT-5.6 Sol pricing 20%. The class-level breakdown isn't confirmed, and both agent stacks in the sources bill via Codex seats, not tokens.
Unsloth Dynamic 3.0 puts its accuracy gains in the small GGUFs
Unsloth Dynamic v3.0 GGUFs for Qwen3.8-27B claim up to +10% top-1 accuracy at equal size, a new imatrix calibration set, and MTP stripped below UD-Q2_K_XL.
What a single-model Metal engine buys you over llama.cpp on Apple Silicon
antirez's h3.c runs MiniMax-H3 on hand-written Metal for Apple Silicon; a macOS VM capability shim made llama.cpp 11.08x faster at prompt processing.
Meta's 30B Muse Glimmer fits a local coding agent under 20GB, Apache 2.0
Meta's Muse Glimmer is a 30B open-weights agentic model under Apache 2.0, quantized to ~4-bit to fit under 20GB with room for KV cache and a drafter.
Other topics
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.