DeepSeek Open-Sources DeepInfer, the Inference Framework Behind Its 155 Trillion Token a Day Service

DeepSeek has open-sourced DeepInfer, the inference framework that powers its production chat service, extending a year-long campaign of releasing the machinery behind its models rather than just the models themselves. The engine is on GitHub under Apache 2.0, together with deepseek-recipe, a production-grade token encoding and decoding library released under MIT that mirrors the logic of DeepSeek's official API.

The company's argument is that model weights set the ceiling of what AI can do, but the inference framework — the long-unexamined middle layer that orchestrates GPU clusters — determines what serving that model costs in compute and electricity. By publishing its internal deployment stack, DeepSeek wants developers and enterprises to replicate the throughput and latency it achieved at scale, in private or public cloud deployments.

The production numbers DeepSeek attached to the release, all company-reported, are the story's real payload: a single cluster serving DeepSeek V4.1 Flash at roughly 155 trillion tokens per day, generating revenue equivalent to 38 dollars per GPU per hour at grid electricity prices. The company says future models' inference code and optimizations will continue to land in the same repository.

Technically, DeepInfer integrates high-performance kernels with a deeply optimized engine supporting distributed inference, speculative decoding, SWA bounded replay and structured output. Hardware support explicitly covers NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs — a dual-vendor stance that has become DeepSeek's signature. The engine builds on a vLLM-derived core with ideas from SGLang and TensorRT-LLM, folding in the company's earlier open-source components including FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect.

The initial release is deliberately scoped. It ships offline inference only; a complete online service still requires distributed KV cache storage, deployment across separate prefill and decode stages, and load balancing across replicas — the architecture is described in an accompanying production report but the orchestration infrastructure is not included. The production configuration separates vision encoding, prefill and decode into independently scheduled stages, with decode running expert parallelism groups of 64 to 128 devices.

For the open-source ecosystem, the release is the most detailed look yet at how a frontier-scale Chinese lab actually serves traffic profitably. Independent labs will now be able to inspect, benchmark and reproduce the engine's execution path — the kind of transparency that is hard to fake and easy to verify.

For the competitive landscape, DeepInfer deepens the moat question for Western inference providers. If a company can publish the exact software that turns raw GPUs into 155 trillion tokens a day at 38 dollars per GPU hour, every cloud's inference margin becomes a benchmarkable claim rather than a marketing one.