On October 20, DeepSeek open-sourced its in-house inference framework DeepInfer together with the production-grade token codec deepseek-recipe — the first time a frontier large-model company has publicly released the native inference framework actually running in production.
By the numbers: DeepInfer already serves DeepSeek V4.1 Flash at scale. A single cluster sustains 155 quadrillion tokens of service throughput per day, with average time-to-first-token of 400 ms, generation at 230 tokens/s (300 in coding scenarios), and a 96% input cache hit rate.
Highlights: distributed inference and load balancing, speculative decoding (DSpark + confidence-head verification, +6% online throughput), SWA Bounded Replay; compatible with both NVIDIA GPUs and Huawei Ascend NPUs.
DeepSeek previously open-sourced its models, the TileLang toolchain, and inference operator libraries; releasing the deployment "middle layer" fills in another piece of China's open-source large-model picture.
Source link: Read the original article. Summary prepared from the original; for reference only.
