Nvidia dominates MLPerf inference tests while AMD sits it out

Every time AI benchmarks drop, the most interesting part isn't always who won, but who didn't even show up.

MLCommons just published its latest MLPerf inference results, and Nvidia once again swept all seven benchmarks. With demand shifting heavily toward generative AI, the suite added two major real-world workloads: Meta's 70-billion-parameter Llama 2 and Stable Diffusion.

Nvidia submitted its upcoming H200 alongside its current lineup, while Intel showed up with its Gaudi2 accelerator and 5th Gen Xeon CPUs. Conspicuously absent once again was AMD and its MI300, alongside other major players like Google and Qualcomm.

Big memory and clever compilers

Nvidia didn't just win on raw silicon muscle; software optimizations did a huge chunk of the heavy lifting.

Over the past six months, Nvidia's open-source TensorRT-LLM compiler delivered nearly a three-fold performance jump on the GPT-J model for the H100. That easily beat Nvidia's initial promise of a two-fold boost.

On the H200, which packs 141GB of high-bandwidth memory, the extra capacity unlocked massive key-value cache optimizations. Nvidia's results showed the H200 outperforming Intel's Gaudi2 by four-fold on inference.

Nvidia also revealed that inference workloads now make up roughly 40% of its data center business, a figure almost certain to grow as enterprise AI moves from training rooms to live products.

Intel holds the line while AMD stays quiet

Intel took full advantage of the thin competition. Since AMD didn't submit, Intel claimed Gaudi2 is currently the only benchmarked alternative to Nvidia's H100.

Intel positioned Gaudi2 on value, arguing it provides comparable or better performance per dollar, lowering total cost of ownership. Intel also showed its 5th Gen Xeon CPUs deliver a 42% AI performance bump over the 4th Gen, keeping them relevant for smaller models.

AMD's decision to skip MLPerf again feels like a missed opportunity. Analysts suspect AMD was simply too swamped tuning workloads for prospective MI300 customers to bother with standardized benchmark runs.

Why it matters

For enterprise buyers and developers, this round of results confirms that inference efficiency depends on memory headroom and compiler maturity just as much as raw FLOPS. If you are deploying 70B models, the extra memory in the H200 makes running them vastly more economical than on an H100.

For the industry, Nvidia revealing that inference accounts for 40% of data center revenue marks an official changing of the guard. The battleground has shifted from model training to production inference, which is where real-world IT budgets will actually be spent.

Intel played this smartly in our view. By submitting results knowing it would lose on absolute throughput against H200, it established Gaudi2 as a legitimate budget-friendly alternative for teams priced out of Team Green.

It seems that the biggest threat to this ecosystem is vendor participation. With AMD, Google, AWS, and Qualcomm sitting on the sidelines, MLPerf risks turning into a curated Nvidia showcase, forcing customers back behind NDAs just to get realistic head-to-head comparisons.