The Verge's Decoder podcast just dug into the uncomfortable reality of AI's massive power appetite. While tech giants like Microsoft and Google blow past their emissions targets to pack data centers with GPUs, high energy use is already standard across video streaming and gaming. The real debate is how we decide if AI is actually worth the juice.
NVIDIA just rolled out full-stack optimizations for Meta's brand-new Llama 3 across its data center GPUs, edge devices, and PCs. In early benchmarks, a single H200 GPU churned out roughly 3,000 tokens per second on the 70B model, enough to handle 300 users simultaneously. The 8B model is also landing on RTX PCs and Jetson Orin modules.
NVIDIA CEO Jensen Huang just told European founders that AI is entering a massive second wave driven by sovereign AI demands and cross-industry adoption. To power this regional push, French cloud provider Scaleway fired up Nabu, an NVIDIA DGX SuperPOD packing 1,016 H100 GPUs designed to keep European data strictly compliant with local privacy laws.
NVIDIA just introduced TensorRT-LLM, an open-source library built to accelerate large language model inference on modern GPUs. It brings in-flight batching and FP8 quantization, delivering up to an 8x throughput boost on H100 GPUs and running Llama 2 4.6x faster than A100 systems. Why it matters: teams can now tap peak GPU parallelism via a modular Python API without writing low-level CUDA, slashing data center running costs.
Google Cloud and Nvidia just announced the general availability of A3 instances featuring Nvidia H100 GPUs. The partnership also brings Nvidia acceleration to Google's PaxML framework, now accessible via the NGC container registry to streamline massive model training. Why it matters: Cloud teams get serious horsepower to tackle demanding generative AI workloads.
