Google Cloud unveils its next-gen TPU with 2x training efficiency

Google is stepping up its hardware game to ease the AI compute crunch.

At its annual Cloud Next conference, Google Cloud unveiled the fifth generation of its custom Tensor Processing Units (TPUs), engineered specifically for large-scale AI training and inferencing.

While Google originally announced TPU v4 in 2021, developers had to wait until 2022 to get their hands on it. With this fifth edition, the company is skipping vanity specs and zeroing in on pure efficiency.

More bang for the buck, scaled across clusters

The main headline here is sheer price-performance.

Compared to the previous generation, the fifth-gen TPU delivers a 2x improvement in training performance per dollar and a 2.5x jump in inferencing performance per dollar.

Mark Lohmeyer, VP and GM of compute and ML infrastructure at Google Cloud, called it the most cost-efficient and accessible cloud TPU the company has built to date.

Scale is the other big unlock. Previously, workloads were hemmed in by physical hardware limits.

Now, Google allows customers to stretch a single AI workload beyond the physical boundaries of a single TPU pod or cluster, scaling across tens of thousands of chips without breaking the bank.

On top of that, Google announced that Nvidia's H100 GPUs will reach general availability next month in its A3 virtual machine lineup.

Why it matters

AI compute is currently the world's most expensive bottleneck. Doubling training performance per dollar and cutting inference costs by 60% changes the unit economics for any team trying to ship generative AI.

Being able to link tens of thousands of chips across physical boundaries is also huge for teams scaling massive foundation models, where single-pod limits often slow things down.

We think Google played this smartly by keeping a two-track strategy. Pushing its custom silicon gives Google crucial leverage and shields it from total Nvidia dependency, while making H100s generally available next month ensures no enterprise customer walks away just because they rely on CUDA.

The real hurdle, as always, is developer inertia. Even with superior cost efficiency, convincing developers to run on TPUs instead of the ubiquitous Nvidia stack takes real effort.