llama-modchip runs 284B DeepSeek-V4-Flash in 64GB RAM at 9.1-14.7 tok/s, no GPU

The repository is synchopate/llama-modchip on GitHub. It is a patched llama.cpp stack bundled with prebuilt binaries, Hugging Face GGUFs and a ./modchip wrapper. It also targets 30B MoEs: the README claims Nemotron-3.5-30B-A3B runs in 14.35 GiB at 25.0-60.4 tok/s on the same 16-core Zen 5 part. For anyone evaluating it, the open question is whether a 284B model compressed this far does useful work at about 10 tokens a second. The sources answer the speed half and leave the quality half largely unmeasured.

The README's starting point is a 76.87 GiB IQ2_XXS file that it says crashes with OOM on 64 GB Linux. Two techniques bring that down. The first is an analytical second-order SwiGLU-H2 Hessian compression of the MoE layers, which the project says shaves 20.31 GiB. The second is Fractal-64 transcoding of the MLA layers into a block-64, BF16-scaled, Gray-coded format called Q2_0 at 3.75 bpw. On top of this, the README lists L2-quadrature MoE pruning and an exact SVD of wo_b * wo_a at 25% lower rank, reported at 99.82% cosine similarity.

The baseline row is internally inconsistent. The README marks the unpatched IQ2_XXS as OOM on 64 GB, yet lists it at 5.91 tok/s decode (169.3 ms). It also marks speculative decoding on that baseline as N/A because of a rollback bug, which the project says it fixed. The sources do not explain on what machine the 5.91 tok/s figure was measured, and they do not give an overall bits-per-weight for the final 56.56 GiB file.

Commenters on Hacker News questioned the baseline itself. One noted that unsloth's IQ2_XXS GGUF of this model is 91GB and its smallest IQ1_S is 82.5GB. That commenter asked which model or quant the 76.87 GiB figure refers to, and the README does not say. The same commenter read "compression" and "MoE pruning" as signs of a model degraded further than unsloth's IQ1 quants. The quality section below covers what the README does and does not measure.

The README breaks out four profiles, all measured on the Ryzen 9 9955HX with 0 GPUs. They share a prefill (pp512) of 43.16-44.36 tok/s. The project credits the 77% gain over 25.0 tok/s to tiling across four levels: ZMM AVX-512BW registers, a 32 KB L1D, a 1 MB L2 and a 64 MB L3. It also cites a register-stationary AVX-512 MQA8 FlashAttention kernel, reported at 2.0x.

Speculative decoding accounts for the gap between the 9-10 tok/s pure decode and the 14.7 headline, and its speedup depends on acceptance rate. The README gives 96.4% acceptance for the Nemotron drafter but no acceptance figure for DeepSeek, so the PVS range should be expected to vary by workload. Prefill is the larger cost for agent use. At 44.36 tok/s, a cold 10,000-token prompt would take roughly 225 seconds to ingest before the first output token. That figure comes from pp512, and the sources do not report prefill speed at longer prompt lengths.

The benchmark table gives probability cosine similarity, KL divergence and top-1/top-3 match for each profile. The reference row, scored 1.00000 and 0.0000 nats, is the standard IQ2_XXS quant. The default profile's "Bit-Exact" label therefore means bit-exact against that IQ2_XXS quantization, not against the released weights. The other profiles drift further from that reference.

The sources report no comparison against BF16 or FP8 DeepSeek-V4-Flash and no task benchmark for the 284B model. The README reports its 100-point quality gate only for the Nemotron 30B profiles. Commenters on Hacker News were skeptical. One said that even a 97GB Q2XS of this model shows very significant divergence from full precision, and that it needs something like 150GB of RAM to run properly with an unquantized KV cache.

Another commenter cited a TUM paper, "Reliability Scaling Laws for Quantized Large Language Models", as evidence that the size-to-performance ratio generally peaks at 4-bit. A reply pointed out that the paper treats quantization-aware training as out of scope. The same reply added that it saw no evidence this project uses QAT, so the paper's findings may still apply here.

The design keeps everything resident: the README states "100% in RAM, zero NVMe swap." No weights are read from SSD during inference, so storage bandwidth affects only load time. The sources give no load time or required read rate. Decode depends on DRAM bandwidth and cache behavior instead. On Zen 5, the project uses _MM_HINT_NTA for victim-L3 isolation of the 64 MB L3, plus 2MB transparent huge pages.

On an 8-channel DDR5 Threadripper PRO 7985WX (Zen 4, 64 cores, 8 CCDs, 256 MB L3), the README reports 111.6 tok/s for the 30B model. It says AVX-512BW register tiling saturates the 8-channel bandwidth there. It recommends -t 32 with a 4-cores-per-CCD stride, which it measured at 54% faster than -t 64 because it avoids Infinity Fabric saturation. That machine is listed with DeepSeek-284B as a recommended model, but no 284B speed is given for it, and no GB/s figures are given for any machine.

Headroom is tight. 58.9 GiB of RSS on a 64 GB box leaves little room for anything else running alongside it. The DeepSeek quickstart command also uses -c 32768, not the advertised 256K context. A separate Tom's Hardware headline describes a homelabber running a 2.8-trillion-parameter model on CPU alone using second-generation Optane memory sticks, with blower fans to keep Optane's heat from burning out nearby DRAM. That is a different memory tier, and only the headline was available.

The Fractal-64 GGUFs use GGML_TYPE_Q2_0 with type ID 40, plus custom deepseek4 and Stockfish-PVS kernels. The README warns that unpatched llama.cpp, Ollama or LM Studio will fail with "unknown tensor type 40". To run them, you need either the ./modchip wrapper or the standalone llama-server-modchip-linux-x86_64 binary, which has zero .so dependencies and ships in the Hugging Face repos and GitHub Release v2.4.0.

This creates a dependency on one project's binary and file format, and the sources give no upstream path. The 284B kernel optimizations the README lists are AVX-512BW. AVX2, Apple Silicon and ARM64 paths are listed for other models and platforms. The verified 284B numbers come from a single Ryzen 9 9955HX. One Hacker News commenter said the README makes it unclear what platform assumptions the stack depends on.

The README targets the prefill problem for agents directly. ./modchip claude points the Claude Code CLI, with all 40 tools active, at the local server. It uses a /dev/shm Slot-0 KV warm restore, so a returning turn restores state instead of re-evaluating the prompt. The project reports 13.1-32.6 ms restores on DeepSeek-284B and 18.9 ms on Nemotron-30B, which it describes as about 6,800x faster warm turns for Nemotron. It frames this as eliminating a 45-second CPU prompt re-evaluation bottleneck.

Warm restore helps only when the prefix is unchanged. New file contents, tool outputs and long diffs still go through prefill at about 44 tok/s, and the sources give no end-to-end agent task timings. For teams considering coding agents on this setup, the measured profile is a box that answers slowly but restores context quickly. The quality of its answers has not been measured against the full-precision model.