NVIDIA supercharges Meta's Llama 3 across all its chips
NVIDIA just rolled out full-stack optimizations for Meta's brand-new Llama 3 across its data center GPUs, edge devices, and PCs. In early benchmarks, a single H200 GPU churned out roughly 3,000 tokens per second on the 70B model, enough to handle 300 users simultaneously. The 8B model is also landing on RTX PCs and Jetson Orin modules.
Meta just dropped Llama 3, and NVIDIA was ready on minute one with full-stack optimizations to make it fly across cloud, edge, and PC.
While training frontier models still grabs the headlines, the real battle right now is making them cheap enough to run in production.
Big iron for training, massive throughput for serving
Meta trained Llama 3 on massive clusters packing 24,576 NVIDIA H100 GPUs, connected via RoCE and Quantum-2 InfiniBand. And Meta is not stopping there, with plans to expand its infrastructure to 350,000 H100 GPUs.
On the serving side, the raw numbers are impressive. A typical user reads at around 10 tokens per second, meaning an interactive bot needs roughly double that speed to feel snappy.
In early testing with the 70B parameter model, a single NVIDIA H200 GPU pushed out roughly 3,000 tokens per second. That is enough bandwidth to handle about 300 concurrent users at once from a single card.
Scale that up to an eight-way HGX server, and you get 24,000 tokens per second, capable of serving more than 2,400 active users simultaneously.
Down to the desktop and robotics
NVIDIA did not just target server racks. The 8B version of Llama 3 is already tuned for local silicon.
On edge hardware, the 8B model hits 40 tokens per second on Jetson AGX Orin and 15 tokens per second on Jetson Orin Nano, making it viable for robotics and interactive agents.
Developers can test Llama 3 in-browser via ai.nvidia.com as a packaged NIM microservice with standard APIs. For enterprises, NeMo handles fine-tuning, while TensorRT-LLM and Triton handle deployment across a target fleet of over 100 million RTX-equipped PCs and workstations.
Why it matters
NVIDIA is sending a very clear message here: they do not intend to cede the inference market to specialized ASICs or in-house cloud chips.
By showing that a single H200 HGX system can serve 2,400 concurrent users on a top-tier open model, NVIDIA is squeezing the unit economics of hosting 70B models down to commodity levels.
For developers, this is immediate leverage. If you run RTX hardware locally, you can spin up capable 8B models without paying API toll booths or leaking sensitive data.
The catch, as we see it, is the tightening software grip. By packaging models inside NIM microservices and TensorRT-LLM runtimes, NVIDIA makes deploying models insanely fast, but migrating away from CUDA becomes even harder than before.
Sources (1)
- blogs.nvidia.com(opens in a new tab)Originalblogs.nvidia.com