DeepInfer: DeepSeek's open-source engine serves V4.1 Flash on SM100 and Ascend 950

The README says the engine was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. Its stated scope is DeepSeek models, not general-purpose serving. In the snapshot the repository had 9.8k stars, 853 forks and 3 commits.

The supported model is large. Artificial Analysis lists DeepSeek V4.1 Flash 1017 at 552B total parameters with 16B active per token. It also lists a 1M-token context window, text and image input, and MIT-licensed weights on Hugging Face. The default checkpoint the engine downloads is deepseek-ai/DeepSeek-V4.1-Flash-1017, and DEEPINFER_HF_MODEL_DIR points it at a local copy instead.

The support matrix is narrow. The README lists no other GPU or NPU families, so Hopper, Ampere, AMD and other Ascend parts are not supported targets in this release. The repository ships separate requirements files for NVIDIA and Ascend. You select the target by exporting DEEPINFER_ACCELERATOR (nvidia or ascend) before running build.sh, and it must stay exported wherever DeepInfer runs.

The scheduler overlaps CPU batch scheduling with device execution. The README calls this a zero-overhead batch scheduler, and it is enabled by default. Two further features target reproducibility rather than throughput.

Batch-invariant inference keeps model results identical across different batch compositions and is enabled with DEEPINFER_BATCH_INVARIANT=1. Deterministic sampling produces identical outputs for identical input tokens and sampling parameters. It requires DEEPINFER_BATCH_INVARIANT=1, DEEPINFER_ENABLE_MTP_VARLEN=0 and a fixed seq_seed passed to SamplingParams. Both features are NVIDIA only. For teams running evals or regression tests against a self-hosted endpoint, this means the same prompt returns the same tokens regardless of what else shared the batch.

The two configurations conflict. Deterministic sampling requires turning MTP varlen off, and DEEPINFER_ENABLE_MTP_VARLEN=1 is the switch for DSpark. DSpark accelerates decoding with draft tokens and uses confidence estimates to schedule verification. DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length, which defaults to 5. In practice you choose between reproducible output and draft-token decoding speedups.

Decoder SWA bounded replay reduces prefill work by replaying only a bounded suffix. The retained suffix must cover the SWA window plus the maximum backtracking length, and the default V4.1 setting keeps 140 tokens in compact mode. Structured outputs constrain generation to a regex, JSON schema or EBNF grammar, passed through so_args in SamplingParams.

The README lists two parallelism modes. Attention layers use Attention DP or balanced CP, and expert layers use MoE EP with EPLB. One commenter on Hacker News asked whether DeepSeek's production deployment combines DeepEP or WideEP with pipeline and tensor parallelism. The README does not answer that: it lists no TP or PP mode and does not describe the deployment shape behind DeepSeek's own service.

The README never mentions quantization. It gives no weight formats, no FP8 or lower-precision serving options and no guidance on fitting the 552B-parameter checkpoint into less memory. Teams planning hardware around a quantized footprint will not find that information here and should treat memory sizing as an open question.

DeepSeek lists three pieces as required for a complete inference service and not included in this release. They are a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode. On disaggregation, the README notes that most of the required engine functionality is already implemented. What is missing is the layer around it.

In operational terms, this release is an engine plus an offline example script, examples/offline_inference.py, rather than a drop-in server. A team that wants an online endpoint for V4.1 Flash has to supply KV cache management, a prefill/decode split and a router across replicas. The sources do not describe how vLLM or SGLang handle those layers for this model. They therefore cannot settle whether staying on one of those projects means less integration work, and that comparison is worth running before committing.

The README's performance section shows prefill and decode charts, and run_sweep.sh reproduces them. Prefill was measured on 4 SM100 GPUs and decode on 64, each within one scale-up domain. The text gives no figures and runs no baseline against vLLM, SGLang or TensorRT-LLM. The 64-GPU decode setup also shows that decode was benchmarked at rack-domain scale, not on a single small node.

Commenters on Hacker News circulated larger numbers from a Zhihu post. The post reports a single-cluster throughput of 155 trillion tokens per day and overall revenue of $38 per card per hour at off-peak pricing. Several commenters used the revenue figure to argue that inference is highly profitable.

Others pushed back. One asked why GPU rental prices are rising while OpenRouter token prices for DeepSeek and other open models are falling, if margins are that wide. Another argued that open inference engines spreading across models and hardware would push inference revenue toward base card cost. None of these figures appear in the repository. The thread did not surface latency, concurrency or cluster-size details, so the figures cannot be translated into a per-GPU throughput expectation for a self-hosted deployment.

The only independent speed figure in the sources is Artificial Analysis's 208.7 output tokens per second for DeepSeek V4.1 Flash 1017 (Max). That measures the model as served by an API, not a benchmark attributed to this engine. Teams choosing between this release and vLLM or SGLang should run run_sweep.sh on their own SM100 hardware against an equivalent configuration of the alternative. The comparison should also count the KV cache and routing layers they would have to add.