Google drops 8th-gen TPUs built for the agent era

Training an AI model and running a swarm of multi-step agents demand completely different hardware physics. Yet for years, the cloud giants tried to muscle through both with the exact same chips.

At Google Cloud Next on Wednesday, Google officially split its custom silicon strategy in two. The search giant unveiled its eighth-generation Tensor Processing Unit, ditching the one-size-fits-all approach for two distinct, purpose-built processors: TPU 8t for massive training, and TPU 8i for low-latency inference.

Co-designed with Google DeepMind, both chips are slated for general availability later this year.

Big iron for training, SRAM for reasoning

Google built TPU 8t with a single goal: turning frontier model training cycles from months into weeks. A single superpod packs 9,600 chips and two petabytes of shared high-bandwidth memory, cranking out 121 ExaFlops of compute. That is nearly triple the pod-level performance of its predecessor, Ironwood.

Raw compute is useless if pods keep crashing. To fix that, 8t targets over 97% goodput with automated hardware recovery. Optical circuit switches and automatic link rerouting patch around dead connections without pausing training jobs. Google says its new Virgo network allows near-linear scaling up to an eye-watering one million chips in a single logical cluster.

TPU 8i, on the other hand, is aimed squarely at the agentic era. When autonomous agents swarm to execute multi-step workflows, even milliseconds of lag compound fast. Google tackled the memory wall by strapping 288 GB of HBM alongside 384 MB of on-chip SRAM, keeping active working sets and KV caches entirely on-chip.

Paired with Google’s custom Axion Arm CPUs and a new Boardfly interconnect topology for MoE models, 8i slashes on-chip latency by up to 5x. Google claims this delivers an 80% improvement in performance-per-dollar over the last generation, letting teams handle nearly double the workload for the same bill.

Citadel Securities is already on board, and both chips support standard tooling like PyTorch, vLLM, and SGLang alongside native bare-metal access.

Yet this is hardly an Nvidia killer. Google confirmed it will still offer Nvidia’s upcoming Vera Rubin systems later this year, and the two are actively collaborating on Falcon networking. As analyst Patrick Moorhead quipped on X, he predicted Google’s original TPU in 2016 would hurt Nvidia, only to watch Nvidia march toward a $5 trillion market cap.

Why it matters

Inference is where the real bills are paid once models hit production. For engineering teams deploying autonomous agents or running heavy token generation, TPU 8i looks like a serious cost cutter. If your stack already runs on vLLM or SGLang, testing 8i later this year is an easy call.

We think Google’s move to bifurcate training and serving silicon is the right architectural bet. Frontier training demands brutal scale and fault-tolerant interconnects, while agentic inference lives and dies by SRAM and memory bandwidth. Trying to satisfy both with a single die simply forces too many compromises.

That said, breaking Nvidia's enterprise gravity remains an uphill battle. The fact that Google must still deploy Vera Rubin proves enterprise customers still want CUDA flexibility. But by coupling custom Arm CPUs, DeepMind co-design, and specialized silicon, Google is steadily proving that hyperscalers can control their own destiny in the agentic era.