DeepSeek open-sources DeepInfer with Huawei chip support
DeepSeek just open-sourced DeepInfer, the production inference engine behind its DeepSeek-V4.1-Flash model. Crucially, the framework supports Huawei Ascend 950 NPUs alongside Nvidia SM100 GPUs, letting teams deploy without Nvidia hardware. Why it matters: It offers a practical path toward diversifying inference silicon, though production serving still requires an external KV cache and load balancing.
DeepInfer is DeepSeek's newly open-sourced offline inference framework, designed specifically to run its DeepSeek-V4.1-Flash model. In plain terms, DeepSeek just pushed the exact high-throughput engine that powers its own production workloads to GitHub under the Apache 2.0 license.
The real headline here is silicon support. Alongside Nvidia SM100 GPUs, DeepInfer provides first-class support for Huawei Ascend 950 NPUs. For teams looking to deploy frontier-class open models without Nvidia hardware, this is one of the most serious turnkey engines made public to date.
What it does
DeepInfer is built to maximize throughput and minimize latency when processing massive contexts and heavy concurrent prompts.
Under the hood, DeepSeek packed in its proprietary kernel library, including FlashMLA, DeepGEMM, DeepEP, TileKernels, and DeepSelect. On the distributed side, it implements Attention DP, balanced CP, and MoE Expert Parallelism with load balancing (EPLB) to keep hardware utilization evenly spread.
To speed up token generation, the engine relies on DSpark speculative decoding paired with confidence-scheduled verification, defaulting to five draft tokens per step. It also features SWA bounded replay for the decoder, slashing redundant prefill work on long sequences by only replaying a strictly bounded context window.
DeepInfer also provides structured outputs via regex, JSON schemas, and EBNF grammars. For Nvidia hardware, it adds batch-invariant inference and deterministic sampling, guaranteeing bitwise identical outputs across dynamic batch compositions when fixed seeds are provided.
Getting it running
This is not a lightweight tool for casual workstations; it targets massive enterprise compute.
On the hardware side, you need either Nvidia SM100 GPUs or Huawei Ascend 950 NPUs. DeepSeek's own benchmark setup runs 4 SM100 cards for prefill and 64 for decode within a single scale-up domain.
Your software stack requires Linux, GCC 13 or newer, and Python 3.12. You will also need CUDA 13 on Nvidia or CANN 9.2 on Ascend. Setup involves setting the DEEPINFER_ACCELERATOR environment variable to either nvidia or ascend, followed by running build.sh. Running the offline script automatically downloads the default DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face.
Keep in mind that DeepSeek explicitly open-sourced an offline inference engine, not a ready-to-serve web cluster. A production-ready online service still requires you to bring your own external KV cache system, Prefill/Decode disaggregation layer, and dynamic load balancing.
Why it matters
DeepSeek openly acknowledges that DeepInfer evolved out of vLLM, with architectural ideas borrowed from SGLang and TensorRT-LLM. The big difference is that DeepInfer ties DeepSeek's proprietary architectural tricks directly into working code for both Nvidia and Huawei silicon.
If your organization has access to Ascend 950 chips or top-tier Nvidia clusters and plans to host DeepSeek-V4.1-Flash locally, this repository is essential reading. However, because it lacks turnkey serving primitives like distributed KV cache management, engineering teams should treat it as a foundational high-performance runtime rather than an immediate drop-in replacement for vLLM or TGI.
Sources (1)
- github.com(opens in a new tab)Originalgithub.com