Nvidia supercharges AI computing with the H200 GPU

Running frontier AI models has created a brutal reality check for data centers: compute isn't the only bottleneck, memory bandwidth is.

At the SC23 conference, NVIDIA took the wraps off the HGX H200, an upgraded beast built on the Hopper architecture that marks the debut of HBM3e memory in an enterprise GPU.

Solving the memory bottleneck

The marquee upgrade here is pure capacity and throughput.

H200 packs 141GB of HBM3e memory delivering 4.8 terabytes per second of bandwidth. Compared to the older A100, that is nearly double the capacity and 2.4 times the bandwidth.

That extra breathing room translates directly into inference performance. NVIDIA says the H200 nearly doubles inference speed on Llama 2 70B compared to the H100.

For massive workloads beyond 175 billion parameters, an eight-way HGX H200 configuration pumps out over 32 petaflops of FP8 compute with 1.1TB of total high-bandwidth memory.

Crucially, server makers do not need to redesign their racks. The HGX H200 is drop-in compatible with HGX H100 systems across both hardware and software, meaning OEMs like Dell, HPE, and Supermicro can roll it out fast.

Systems are slated to ship in the second quarter of 2024, with major cloud players like AWS, Google Cloud, Microsoft Azure, and Oracle Cloud ready to deploy.

Why it matters

We think NVIDIA is pulling a textbook mid-cycle refresh that directly targets the biggest operational headache in generative AI today, which is inference cost.

For developers and teams serving production workloads, doubling throughput on models like Llama 2 means effectively halving the cost per token, without needing to rewrite software stacks. That said, it will not hit production until Q2 2024, so anyone desperate for compute right now still has to battle the existing H100 shortages.

On the competitive front, this puts immense pressure on rivals trying to challenge the original H100. By keeping the exact same board footprint, NVIDIA makes it almost trivial for cloud providers to keep signing checks rather than testing alternative silicon.

The real risk here comes down to supply chain execution. Packaging HBM3e at volume is notoriously tricky, and whether NVIDIA can deliver enough units to satisfy skyrocketing demand remains the big question.