Google debuts Ironwood, its first TPU built specifically for inference

Training frontier AI models used to be the biggest money pit in tech. Now, running them is.

With reasoning models that pause to think, verify, and generate chains of thought before answering, inference compute demands are going through the roof. At Google Cloud Next 25, Google unveiled its countermove: Ironwood, its seventh-generation custom TPU and the first one explicitly designed for the "age of inference."

Slated to arrive in Google Cloud later this year, Ironwood takes dead aim at the memory bottlenecks holding back complex reasoning tasks.

Solving the memory bottleneck

Reasoning models, mixture-of-experts (MoE) architectures, and agentic workflows do not just need raw math power; they need massive memory bandwidth so data doesn't get stuck in traffic.

Google packed each Ironwood chip with 192 GB of High Bandwidth Memory (HBM). That is a massive 6x jump over last year's sixth-generation Trillium chip. The memory bandwidth reaches 7.37 TB/s per chip, representing a 4.5x boost.

Inter-chip interconnect (ICI) bandwidth climbed 1.5x to 1.2 TB/s bidirectional. The silicon also features an upgraded SparseCore accelerator designed for ultra-large embeddings, useful for advanced search ranking and recommendation models.

Pods scaling past supercomputers

For enterprise workloads, Google plans to offer Ironwood in pods of 256 chips or full-scale liquid-cooled clusters of 9,216 chips.

A single 9,216-chip pod delivers 42.5 Exaflops of peak compute while consuming nearly 10 MW of power. For context, Google notes that exceeds the compute capacity of El Capitan, the world's most powerful supercomputer, by more than 24 times. Each individual chip provides 4,614 TFLOPs of peak compute.

Despite the power density, efficiency gains are substantial. Google claims Ironwood delivers twice the performance per watt of Trillium, and sits at nearly 30 times the energy efficiency of Google's first Cloud TPU from 2018. Sundar Pichai added during his keynote that performance has jumped 3,600 times compared to their first public TPU.

Google DeepMind's Pathways software stack will manage distributed workloads across pods, allowing developers to compose hundreds of thousands of chips together.

Why it matters

We think Ironwood reflects a decisive pivot in hyperscaler infrastructure: the center of gravity has shifted from training runs to continuous inference at planetary scale.

With Google embedding Gemini models across 15 products that each have over half a billion users, relying entirely on off-the-shelf merchant silicon would be financially ruinous. By heavily tailoring Ironwood's memory capacity to 192 GB, Google makes it viable to serve massive thinking models and MoE layers without fracturing them across countless memory-starved accelerators.

For enterprises and developers, this could unlock cheaper and faster long-context reasoning inside Google Cloud later this year. The catch, however, remains ecosystem lock-in. Running on Ironwood requires embracing Google's proprietary Pathways stack and TPU environment, creating deep gravity that keeps customers tightly tethered to Google Cloud.