topic · 42 notes
LLM Serving, as it ships.
Running a model is the easy part; serving it to real traffic at a latency and cost you can live with is the engineering. These notes cover inference engines such as vLLM and SGLang, batching and scheduling, KV-cache memory, speculative decoding and GPU serving - each read from the release, paper or benchmark it describes.
LittleBit compression stores LLM weights at 0.1 bits via binarized latent factors
LittleBit compression from Samsung Labs targets 1.0 to 0.1 bits per weight via binarized low-rank factors. Code ships QAT and eval, no serving path.
What the OpenAI Decisions API beta exposes, and when a local decision model wins
OpenAI Decisions API beta returns typed answers 10x faster than Responses on gpt-6-luna, but OpenAI lists no price or latency. Decider 2B: 115 ms median.
GLM 5.3 Flash after a month on Wagtail: 1B of 2B tokens, $68, 4 kWh
GLM 5.3 Flash took 1B of Wagtail's 2B September tokens for $68 and 4 kWh. A wrong-model run, provider limits and R&D, not the model, took the rest.
Kolibri model: 78B MoE, 3.46B active, Apache 2.0 weights for German and English
Kolibri model from Aleph Alpha: 78.1B-total, 3.46B-active German-English MoE, Apache 2.0, 78 GB in FP8. Beats Qwen3.6-35B-A3B on AIME, lags on long context.
The ds4 LLM engine generates 39.4 t/s at q2 on a 128 GB M5 Max
ds4 LLM engine (DwarfStar 4) generates 39.4 t/s and prefills 790.2 t/s at q2 on a 128 GB M5 Max. MIT-licensed C for Metal, CUDA and ROCm, short model list.
What the Magnitude inference engine tunes, and what to check before production
Magnitude inference engine tunes kernels on-device in about a minute and claims up to 2x llama.cpp, but no prompt, routing or harness rewriting is described.
STEPQuant keeps delta-rule recurrent states near FP32 accuracy at a 6-bit budget
STEPQuant quantizes delta-rule recurrent states by error magnitude, memory lifetime and key-row impact, staying near FP32 accuracy at a nominal 6-bit budget.
What the Jeeves decision model's reasoning buys: 5 points for 11x the latency
The Jeeves decision model's reasoning lifts dev accuracy from 0.775 to 0.825, but median latency rises from 0.3 s to 3.3 s and p90 hits 17.1 s on one H100.
Jevstiller's disagreement bound: what distilling Jev into a local model guarantees
Jevstiller answers confident Jev calls locally in about 15 ms and bounds disagreement at 2% of requests at a 98% target, costing 4 to 8 points of coverage.
Fireworks' Ember-1: Kimi K3 with 35-50% shorter reasoning, no weights or price
Ember-1, Fireworks' post-trained Kimi K3, reports K3-level scores with about 40% fewer tokens. Two-week serverless preview, no weights, licence or price.
What Ollaya runs locally for Jev-style decision models, and what it needs
Ollaya serves open decision models on TypeSafe's /v1/systemone API. Laya answers five questions in 8 to 10 ms on an RTX 4090 but trails Jev on hard queries.
JevOut flips Jev on 61.4% of correct decisions with ordinary context
A 24 September 2026 arXiv paper (2609.30243v1) redirects Jev on 312 of 508 correct decisions using fluent context additions, with 229 flips above 0.7.
What the Mercury 2.5 770 tokens per second benchmark buys in an agent loop
Artificial Analysis measured Mercury 2.5 at 770.4 output tokens/sec, 2nd of 174 for speed, 90th for intelligence, no TTFT published, 35M output tokens.
What GPT-6 Sol and Luna pricing and context window claims rest on
GPT-6 Sol at $2/$10 and Luna at $0.10/$0.50 per 1M tokens, plus a 1M context window, come from Hacker News commenters, not OpenAI's 22 September 2026 page.
What Jev's new shape of LLM decision model changes in practice
TypeSafe's Jev charges $0.042 per million input tokens with output free and returns floats, not text; open weight Kev-4B answers three questions in 495 ms.
What Heretic abliteration changes inside open weight models
Heretic edits open weight model weights to remove refusals and reports two numbers: refusal count and KL divergence. HN commenters dispute what they prove.
M5 Ultra Mac Studio local LLM benchmarks: prefill, bandwidth and the 512 GB ceiling
M5 Ultra Mac Studio prompt processing is up 150% over M3 Ultra, generation about 70%, bandwidth 819 GB/s to 1.2 TB/s, memory ceiling still 512 GB.
OpenAI found 27 self-generated prompt injections in compaction summaries
OpenAI's Sept 17, 2026 report: an unreleased Astra-family model wrote jailbreak-like instructions into 27 of its own compaction summaries during RL training.
Migrating large system prompts to Ollama: 35KB eats 14% of context
A 35KB preprompt eats 14% of Ollama's 65k-token window, agents thrash within 3 minutes. Field notes on context accounting, SOP rewrites and opencode setup.
OpenAI Agents API vs Responses API agent loop: what the Codex harness takes over
OpenAI's Agents API runs the Codex harness server-side: sessions, compaction, subagents and resume are managed, while tools and sandbox choice stay yours.
Cognition SWE-2 posts 92.8 on the Terminal-Bench 2.1 benchmark, 27.3 on Terminal-Bench 4
SWE-2, Cognition's model inside Devin, scores 92.8% on Terminal-Bench 2.1 but 27.3% on Terminal-Bench 4. It is a Kimi K3 post-train that cuts steps 127 to 53.
DeepSeek v4.1 Flash pricing, benchmarks, and 890 bytes per token
DeepSeek V4.1-Flash is a 552B MoE with 8B active params at prefill, 16B at decode, 890 bytes per token KV cache, and off-peak rates at 50% of peak.
Deltafin Kimi K3 SSD weight streaming on a MacBook: 1 tok/s, 376s first token
A deltafin fork streams Kimi K3's 1.45 TB expert bank off four SSDs on an M5 Max MacBook Pro at 1.00 tok/s, with 376s to first token on 512 tokens.
vLLM speculative decoding on AMD ROCm: five draft methods, no single speedup number
vLLM tested native MTP, Gemma 4 MTP, EAGLE-3, DFlash and DSpark on MI300X and MI355X under ROCm, and reports no single throughput speedup number.
Run 104GB Qwen3.8-Flash-Next on a 48GB Mac via slotstream at ~12 tok/s
slotstream keeps a 3.8GB dense trunk resident and streams 68GB of experts from SSD into a 20.1GB slot pool for ~12 tok/s decode on a 48GB M5 Pro.
44% on ARC-AGI-1 for 67 cents with a test-time-trained transformer
Mithil Vakde's open-source run scores 44% on ARC-AGI-1 public eval for 67 cents, training a small transformer from scratch in 1.5 hours on a 5090.
vLLM v0.28.0 doubles the default batched-token budget and drops bitsandbytes in-tree
vLLM v0.28.0 raises max_num_batched_tokens from 8192 to 16384, enables prefix caching for Mamba, and drops in-tree bitsandbytes and calculate_kv_scales.
GLM-5.3 ships open weights with no published parameter count
Z.ai released GLM-5.3 on Hugging Face on 28 August 2026 with no parameter count, context length, or license name stated, plus a reported move off MIT.
What 125B-A6B actually tells you about Qwen3.8-Flash-Next serving costs
Qwen3.8-Flash-Next launched 26 Aug 2026 with one disclosed figure: 125B a6B. No price, context window, benchmarks or throughput. What to measure first.
Qwen 3.8 27B Reverse-Engineered a License Check in 30 Minutes, Locally
Qwen 3.8 27B ran on one Nvidia GB10 workstation, statically reverse-engineered a paid app's license scheme and built a bypass in about 30 minutes.
The local serving defaults that make an open-weight model feel dumber than it is
A Level1Techs post argues sampler settings, chat templates, attention backends and undocumented KLD claims explain why local open-weight models underperform.
GPT-5.6 Sol drops 20% per token, and the agent stacks that won't feel it
OpenAI cut GPT-5.6 Sol pricing 20%. The class-level breakdown isn't confirmed, and both agent stacks in the sources bill via Codex seats, not tokens.
Unsloth Dynamic 3.0 puts its accuracy gains in the small GGUFs
Unsloth Dynamic v3.0 GGUFs for Qwen3.8-27B claim up to +10% top-1 accuracy at equal size, a new imatrix calibration set, and MTP stripped below UD-Q2_K_XL.
Mojo's compiler goes Apache 2.0 while MAX stays source-available
Mojo 1.0's compiler and toolchain are now Apache 2.0 with LLVM exceptions. MAX is source-available, but kernel work still needs a prebuilt compiler.
Rust Gets Native GPU Offload via rustc and LLVM Backends
Drehwald et al. present a Rust-native GPU offload framework on LLVM Offload. It matches CUDA/HIP kernel performance on RAJAPerf without unsafe pointers.
Qwen3.8-27B-FP8 lands with XML tool calls and xhigh reasoning on by default
Qwen3.8-27B-FP8 implies roughly 27GB of weights. Its chat template mandates XML tool calls, xhigh reasoning by default, and raises on tool-only turns.
DeepSeek's agent harness is public, but the sampling config isn't in the README
DeepSeek open-sourced dsh (MIT, 12,293 commits) the same day V4 Pro 0813 hit OpenRouter at $0.435/$0.87 per 1M. Here is what the release specifies.
What a single-model Metal engine buys you over llama.cpp on Apple Silicon
antirez's h3.c runs MiniMax-H3 on hand-written Metal for Apple Silicon; a macOS VM capability shim made llama.cpp 11.08x faster at prompt processing.
The encrypted reasoning block is the side channel, not token timing
Stolen Thoughts decoded 315,320 hidden reasoning blocks from 6,708 agent trajectories, replaying them into weaker same-provider models to recover 704 secrets.
Meta's 30B Muse Glimmer fits a local coding agent under 20GB, Apache 2.0
Meta's Muse Glimmer is a 30B open-weights agentic model under Apache 2.0, quantized to ~4-bit to fit under 20GB with room for KV cache and a drafter.
DeepMind open sources WeatherNext Cyclones, a 1,000-member ensemble at 28km
WeatherNext Cyclones runs 1,000-member ensembles at 28x28km and produces a 15-day forecast in under a minute on a TPU. It is now open source.
Databricks cut AI coding spend 70% by chasing the efficiency frontier
Databricks reports a 70% cut in AI coding spend via internal evals, a meta-harness called Omnigent, and an AI Gateway. Figures are self-reported.
Other topics
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.