Nvidia unveils Groq 3 LPU for multi-agent workloads

Nvidia just showed off the Groq 3 language processing unit at GTC 2026, marking the first real hardware drop from its massive $20 billion Groq acquisition deal.

Back in December, Nvidia agreed to license Groq's tech and hired founder Jonathan Ross and president Sunny Madra. Just three months later, the chip is already on the floor in San Jose.

While attendees were mostly hyping Nvidia's next-gen Rubin GPU, this specialized inference beast might actually change how multi-agent AI works.

Built for agents talking to agents

Nvidia GPUs are general-purpose powerhouses that handle both training and inference. Groq chips, on the other hand, do one thing and one thing only: lightning-fast inference.

According to Ian Buck, Nvidia's VP of hyperscale and HPC, GPUs may hold much more memory, but Groq 3's memory is dramatically faster.

Nvidia packaged the new silicon into the Groq 3 LPX rack. It packs 256 Groq 3 LPUs, 128GB of solid-state RAM, and delivers a staggering 40 petabytes per second of memory bandwidth.

The real target here is agentic AI. Buck pointed out that 100 tokens per second is plenty fast for a human, but it feels glacial when software agents are constantly talking to each other behind the scenes.

Groq 3 aims to push throughput up to 1,500 tokens per second specifically to grease the wheels for real-time agent-to-agent chatter.

The coprocessor for trillion-parameter models

Groq 3 is not meant to replace GPUs. Instead, Nvidia wants it to work side by side with the new Vera Rubin NVL72 rack, which integrates Rubin GPUs and Vera CPUs.

Nvidia describes Groq 3 as a coprocessor for Rubin. It boosts performance on every token across every model layer, geared directly for trillion-parameter models and million-token context windows.

Paired together, Nvidia claims the two racks pump out 35 times higher throughput per megawatt and create a 10 times greater revenue opportunity.

It was part of a five-rack blitz at GTC, which also introduced a dedicated Vera CPU rack, the Bluefield-4 STX storage rack, and the Spectrum-6 SPX networking system.

The hardware surge comes as Nvidia's data center revenue hit $193.5 billion in fiscal 2026, up from $116.2 billion the year before. With Big Tech hyperscalers like AWS, Google, Microsoft, and Meta expected to pour $650 billion into data centers this year, Nvidia wants to capture every inch of that infrastructure.

Why it matters

For engineers building multi-agent setups, the biggest headache has always been compounding latency. When five different agents need to debate a problem before giving an answer, traditional GPU inference grinds workflows to a halt. Delivering 1,500 tokens per second makes real-time agent workflows actually practical.

We think this move shows just how ruthlessly Nvidia defends its turf. Instead of letting ASIC startups chip away at its inference business, Nvidia absorbed Groq's best tech and turned an existential threat into an upsell.

The architecture looks brilliant on paper: keep the massive memory capacity on the Rubin GPUs, and offload the bandwidth-choked token generation to Groq's blazing SRAM.

However, the potential catch is flexibility and cost. With just 128GB of SRAM per 256-chip rack, Groq 3 cannot survive on its own for heavy enterprise workloads without leaning on an ultra-expensive Rubin cluster. Whether mid-tier cloud customers can justify running this dual-rack beast remains an open question.