Blackwell Ultra crushes MLPerf inference records

AI models are getting a lot smarter, but their thinking process is getting brutally expensive. When a model like DeepSeek-R1 pauses to generate internal reasoning tokens before answering, servers melt and token bills skyrocket.

MLCommons released the official MLPerf Inference v5.1 benchmark results. Making its official debut in the competition, NVIDIA brought out its Blackwell Ultra architecture inside the massive GB300 NVL72 rack system and broke every major inference record in sight.

Demolishing reasoning benchmarks

This MLPerf round updated its testing suite to match modern workloads. Out went legacy tests like GPT-J and RNN-T, replaced by DeepSeek-R1 (a massive 671B MoE reasoning model), Llama 3.1 405B Interactive, Llama 3.1 8B, and the Whisper speech recognition model.

NVIDIA's GB300 NVL72 dominated the board. Running DeepSeek-R1 in the offline scenario, each Blackwell Ultra GPU delivered 5,842 tokens per second. That is a 45% jump over the previous GB200 Blackwell system, and roughly 5x the per-GPU throughput of Hopper-based DGX H200 systems.

Hardware upgrades helped, of course. Blackwell Ultra packs up to 288GB of HBM3e memory per GPU, alongside 1.5x higher peak NVFP4 compute and double the attention layer compute compared to base Blackwell.

Squeezing MoE into 4-bit precision

Raw silicon only tells half the story. To get DeepSeek-R1 running this fast without dropping required accuracy, NVIDIA applied heavy compression tricks using its TensorRT stack.

The base DeepSeek-R1 FP8 weights were quantized down to NVFP4, NVIDIA's proprietary 4-bit floating point format. Even the KV cache, normally stored in BF16, was compressed down to FP8 to save precious memory bandwidth.

Because DeepSeek-R1 is a mixture-of-experts model, standard parallelism broke down. NVIDIA split expert parallelism for the MoE layers and data parallelism for attention, using a custom load-balancing technique called ADP Balance to keep all 72 GPUs equally busy. They also leaned on disaggregated serving for Llama 3.1 405B, separating the prefill context phase from token generation to score a 47% boost over standard setups.

Why it matters

Inference cost is rapidly becoming the single biggest bottleneck in generative AI. As models think longer, running reasoning chains at scale turns into an economic nightmare unless throughput jumps dramatically.

For enterprise teams and developers serving models, Blackwell Ultra effectively slashes the cost per reasoning token. When cloud providers like Azure, CoreWeave, and Oracle spin up GB300 clusters, hosting heavyweight open models like DeepSeek-R1 and Llama 3.1 405B will finally become commercially viable for interactive apps.

Strategically, NVIDIA is widening its moat by shifting the battle from raw FLOPS to full-stack systems engineering. By fusing proprietary NVFP4 formats, NVLink fabric, and deep software optimizations, it makes switching to competing chips painfully disadvantageous. The catch, however, is clear: leaning entirely into NVIDIA's bespoke formats locks customers even deeper into its proprietary walled garden.