DeepSeek has open-sourced DeepInfer, the high-performance inference framework that powers its production services and serves as the engine behind DeepSeek-V4.1-Flash. Released under the Apache 2.0 license alongside the MIT-licensed deepseek-recipe token encoding and decoding layer, this release provides developers with a rare look at a frontier AI lab’s real serving stack. The framework runs on NVIDIA SM100 Blackwell GPUs and Huawei Ascend 950 NPUs, enabling production-grade throughput where a single cluster can serve 155 trillion tokens per day.
Built initially on vLLM with architectural ideas from SGLang and TensorRT-LLM, DeepInfer integrates custom kernels such as FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and TileLang. It introduces advanced optimizations like Attention Data Parallelism combined with Wide Expert Parallelism, confidence-scheduled speculative decoding via DSpark, and SWA bounded replay that keeps sliding-window attention recomputation costs constant regardless of context length. These engineering decisions deliver significant performance, achieving over 25,000 output tokens per second per GPU at 256 requests per rank during benchmarks using 4 GPUs for prefill and 64 for decode.
Despite its impressive speed and efficiency, DeepInfer currently supports offline inference only and lacks built-in components like a KV cache system, prefill-decode disaggregation, and load balancing across replicas. This release matters because it gives the broader developer and research community direct access to the battle-tested infrastructure driving state-of-the-art large language model deployments at scale.