Run 104GB Qwen3.8-Flash-Next on a 48GB Mac via slotstream at ~12 tok/s

slotstream, published on 2026-09-01 by carloslfu, runs Qwen3.8-Flash-Next - a 125B-parameter mixture-of-experts checkpoint that is 105 GB on disk at 4-bit - on a 48 GB M5 Pro at roughly 12 tok/s warm decode with a 32 GB peak. Only the 3.8 GB dense trunk stays resident; the 68 GB of routed experts (512 per layer, 10 active per token) are read from SSD with pread into a fixed pool of cache slots shared by all 48 layers. Engine start is about 2 seconds because only the trunk loads. The constraint that replaces RAM is disk: Apple Silicon, macOS 14 or later, and roughly 110 GB free, which the README says makes a 512 GB Mac the realistic minimum whatever your memory.

The project is one Swift binary with no Python. It listens on port 11434 and implements the chat and generate subset used by Ollama clients and OpenAI SDKs, with Open WebUI and the OpenAI SDKs tested against that subset; unsupported features (tools, images, JSON-schema output, logprobs) return a 400 rather than being ignored. The code is MIT, with GatedDelta.swift ported from mlx-swift-lm (MIT); the weights come from pipenetwork/Qwen3.8-Flash-Next-MLX-4bit and remain under the Qwen community license.

The one-time download is 105.3 GB across 25 files. pull opens eight TCP connections and measured 112 MB/s on a 1 Gbit/s datacenter link, 16 minutes end to end; the README projects about 2 hours 20 minutes at 100 Mbps and about 9 hours at 25 Mbps, and notes that through 0.2.0 it ran on a single connection whatever the flag said. Interrupted pulls resume, and all 25 files are checked against sha256 hashes compiled into the binary. Releases are built by CI from the tagged commit with signed provenance, so an asset can be verified with gh attestation verify instead of trusted.

Memory-mapping the weight files does not work here because MLX cannot materialize part of a memory-mapped tensor. A top-10 expert gather evaluates all 512 experts of that layer, so an mmap path loads roughly 100 GB and dies; the stock mlx_lm.load() route took the same 48 GB machine into 48 GB of swap without producing a token. slotstream reads experts with pread into a fixed slot pool instead, so hot layers borrow slots from cold ones and the pool size is what the memory knobs control.

On a 48 GB Mac at the default 33 GB target, the printed plan allocates about 152 of 512 experts per layer as 7,280 global slots totalling a 20.1 GB pool, budgets about 5.3 GB for prefill at 4096 tokens per pass, and about 1.2 GB for reuse across conversations, expecting a 32.0 GB peak. Cache size changes speed and not output: greedy decoding is byte-identical between a 4 GB cache and a 24 GB one, and that equivalence is a standing test. Output is also byte-identical across the live resizes described below.

Auto-sizing takes the lowest of three limits - a 33 GB cap, 70% of RAM, and 2 GB under the Metal working-set limit - and sizes down further while other apps are holding memory. The 33 GB cap is the knee of the measured curve: in a GB-at-a-time sweep, nothing between 34 GB and 84 GB decoded or prefilled any faster, so a 128 GB Mac gets the same plan a 48 GB one does. While running, slotstream re-checks every 15 seconds and resizes the cache between requests. You can override with --memory-gb (minimum 8.1, and it will go past 33 if you want to experiment), --max-ram-percent, or --experts-per-layer and --pool-gb.

Only the 48 GB row is measured on real hardware, an M5 Pro. The other rows come from slotstream doctor --sim-ram N, which reproduces the curve, and the README notes that smaller Macs also have slower SSDs. Anyone planning to deploy on a 16 GB or 24 GB machine should treat those figures as targets to validate rather than as measurements.

The model ships a draft head, the 1.5 GB mtp.safetensors that pull fetches as the last of the 25 files, which predicts the token after next. With --mtp (default auto), slotstream drafts the next token and verifies it in one two-token pass, and the draft is right 86% of the time. Measured on a quiet 48 GB Mac at a 28 GB target: x1.24 decode, 10.3 to 12.8 tok/s, x1.33 on a code prompt, and x1.18 with the server's default sampling.

The threshold is what decides it. The head costs 1.6 GB out of the cache budget, so auto enables it only when the cache still reaches 120 experts per layer afterwards - a 28 GB target - and keeps it off below that, where it measured a loss; with it on, the auto ceiling becomes 34.6 GB. Without the file, everything runs as before and speculative decode is off. The stronger result on a code prompt matches what Baseten's Philip Kiely describes for speculative decoding generally, that the gains are strongest on code generation where output token sequences are relatively predictable.

The whole prompt is processed before the first token appears, at 4096 tokens per pass and roughly 125 tok/s on the 48 GB machine. An 8,000-token prompt therefore waits about a minute on a 48 GB Mac and over three minutes at a 16 GB target. Prompt plus completion is capped at 32,768 tokens via --max-context.

Within a conversation that cost is paid once: follow-up turns prefill only what is new, and over eight turns at a 16 GB target the last turn took 6.0 s instead of 25.8 s. Reuse covers up to 32,768 tokens across four conversations for about 1.2 GB. Reused state is not bit-identical to recomputing it, so a reply can occasionally differ where two tokens were nearly tied; --no-prefix-cache turns it off where exact reproducibility is required.

The decision between streaming a large MoE from SSD and running a smaller quant that fits in RAM turns on decode rate, prefill rate, and what the smaller checkpoint gives up. Kevin Lewis, writing at lws.io on the same day, reports an M4 Pro Mac mini with 48 GB running Qwen3.6-35B-A3B-OptiQ-4bit entirely in unified memory at 34 tok/s generation and 325 tok/s prompt processing, using about 20 GB of RAM and leaving 28 GB for context and the OS. Against slotstream's ~12 tok/s decode and ~125 tok/s prefill on a comparable-capacity machine, that is roughly 2.8x on decode and 2.6x on prefill, with no SSD reads in the decode loop; the M4 Pro's 273 GB/s of unified memory bandwidth is the ceiling on that path.

What the sources do not supply is the other half of the comparison. Neither the slotstream README nor the lws.io post reports a quality benchmark for Qwen3.8-Flash-Next at 4-bit against Qwen3.6-35B-A3B-OptiQ-4bit, so the capability side of the tradeoff is unmeasured in this material. Baseten's framing is useful for how to settle it locally: efficient frontiers in inference are jagged, small configuration changes have outsized effects, and the cutoff points have to be discovered empirically through sweeps - which is exactly what the 34-to-84 GB sweep and the 120-experts-per-layer MTP gate are.

Commenters on Hacker News named mlx-moe-offload, streamlx, mlx-moe, mlx-flash, and deepseek-v4-flash-mlx as repos already doing the same thing: keep the resident parts of an MoE in unified memory and page or stream routed experts from SSD on Apple Silicon. The objection was fragmentation - that benchmarking against those projects or upstreaming into MLX and MLX-LM would be worth more than another near-identical repo. A reply in the thread said the project is an experiment in seeing how fast this can be made, that its author is committing to maintain it, and that a from-scratch take can surface approaches an existing project has already baked in.

One commenter doubted that 16 GB of unified memory yields 5 tok/s without ignoring thermal warnings, and reported 7-8 tok/s from Qwen3.6-35B-A3B on a 16 GB M3 with optimizations applied; note the README's 16 GB row is ~4 tok/s and is simulated, and thermals are not addressed anywhere in the source material. Another wrote that 12 tok/s is a long way from comparable to Claude once the price of high-RAM Apple Silicon is counted. A third argued 32 GB is too tight in general, allowing 8 GB for the OS plus 4-8 GB for the model and KV cache.

slotstream is measured on one machine, an M5 Pro with 48 GB; the 8, 16, 24 and 32 GB rows are simulated. v0 runs exactly qwen3.8-flash-next:4bit and pull knows no other name, with one model process per user. macOS 14 and 15 have had the installer exercised but not the runtime, and the Ollama CLI cannot connect in 0.2.0 because its requests carry empty name, system, template and options fields plus an empty-prompt load request that the strict validator rejects; the fix is on main and ships in the next release. The README also does not say how the model's 32 GB n-gram table is served, reports no SSD endurance figures for sustained pread workloads, and contains no benchmark against mlx-moe-offload or the other streaming repos.

The quantization question is open in a specific way: no task benchmark for Qwen3.8-Flash-Next at 4-bit appears in any of the sources. Adjacent work published the same day (arXiv 2609.01587v1) measured where post-training quantization damage lives across 9 open-weight models in 4 architecture families by raising each layer to 8-bit in turn, and found recovery diffuse - for 8 of 9 models, recovering 75% of the gap takes roughly half the layers - and that at a matched precision budget, spending it globally on finer quantization granularity beat locally repairing the most recoverable layers by 21-52 points for all 8 group-128-compatible models. That study does not include Qwen3.8-Flash-Next, so it sets expectations about how to spend a bit budget rather than telling you what this checkpoint loses.