NVIDIA's Helix brings 32x more users to million-token AI inference

Ask an AI model to read through an entire encyclopedia, months of chat history, or a sprawling codebase, and you'll quickly hit a wall. Generating text in multi-million-token contexts is painfully slow.

Every time the model produces a new token, it has to stream that massive historical context (the KV cache) out of DRAM, while also loading heavy Feed-Forward Network (FFN) weights. These two bottlenecks choke memory bandwidth. Even worse, traditional Tensor Parallelism falls apart with modern architectures like Llama's GQA or DeepSeek's MLA, forcing GPUs to duplicate the multi-million-token cache across multiple cards.

NVIDIA introduced Helix Parallelism, a new hybrid sharding strategy co-designed with its Blackwell systems to crack real-time decoding for multi-million-token contexts.

Decoupling attention and FFN

Inspired by the structure of a DNA helix, Helix refuses to treat a Transformer layer as a static pipeline. Instead, it decouples the parallelism strategies for the attention phase and the FFN phase, reusing the exact same pool of GPUs for both.

During attention, Helix applies KV Parallelism to slice the multi-million-token cache along the sequence dimension across GPUs, keeping Tensor Parallelism within the limit of KV heads to prevent memory duplication entirely. Each GPU computes local FlashAttention, followed by a single all-to-all communication to swap partial outputs. Because this exchange scales with batch size rather than sequence length, it stays lightning-fast even as context balloons into millions of tokens.

To shave off even more latency, Helix overlaps communication and computation across the batch (HOP-B). While one request runs its all-to-all exchange, the next request is already computing attention.

Once attention wraps up, that same set of GPUs immediately reprovisions without idle time into a Tensor Parallelism layout for dense layers, or a 2D grid combining Tensor and Expert Parallelism (MoE) for models like DeepSeek. To prevent memory hotspots, Helix even writes new tokens to the KV cache in a round-robin schedule across cards.

32x the users at the same latency

NVIDIA's exhaustive simulations on Blackwell systems—leveraging FP4 compute and high-bandwidth NVLink 72—showed massive leaps on DeepSeek-R1 (671B) decoding a 1-million-token context.

At a fixed latency target, Helix boosted concurrent user capacity by up to 32x, translating directly to a 32x increase in tokens per second per GPU.

In low-concurrency settings where raw interactivity matters most, it slashed the minimum token-to-token latency by up to 1.5x.

Why it matters

Long-context AI has suffered from an ugly trade-off: it exists on paper, but serving it interactively to paying customers has been prohibitively expensive. Helix directly tackles the unit economics of real-time multi-million-token agents. If you can serve 32 times more users on the same cluster, whole-codebase copilots and autonomous legal assistants suddenly become viable commercial products.

It also highlights NVIDIA's true moat. Helix isn't just clever math; it relies heavily on the massive interconnect bandwidth of the NVL72 domain to make low-overhead all-to-all swaps possible. Rivals can try to match NVIDIA on raw FLOPS, but without massive rack-scale bandwidth, implementing this level of dynamic sharding is nearly impossible.

That said, it is worth remembering these numbers stem from architectural simulations rather than public production clusters. We will be watching closely to see how well these theoretical gains survive real-world, unpredictable traffic spikes once NVIDIA bakes Helix into standard inference runtimes.