Microsoft's custom Maia 200 chip takes on AI inference

Hyperscalers have poured billions into Nvidia's pockets to get the generative AI boom off the ground, but turning millions of daily users into a profitable business requires slashing the cost of every single token. On January 26, Microsoft revealed its latest secret weapon to do just that: the Maia 200 inference accelerator.

Built on TSMC's 3nm node, Maia 200 packs over 140 billion transistors into a 750W SoC. It is specifically tuned for narrow-precision computation, churning out over 10 petaFLOPS of 4-bit (FP4) performance and over 5 petaFLOPS of 8-bit (FP8) performance.

Microsoft isn't shy about comparing it with rivals either. The company claims Maia 200 delivers three times the FP4 throughput of Amazon's third-generation Trainium, edges past Google's seventh-generation TPU in FP8 workloads, and offers 30% better performance per dollar compared to the latest generation hardware in its existing Azure fleet.

Beating the memory wall with standard Ethernet

Pure compute power means nothing if your chip spends half its time waiting for weights to load. Maia 200 attacks this classic memory bottleneck with 216GB of HBM3e running at 7 TB/s, supported by 272MB of on-die SRAM, dedicated DMA engines, and a custom Network-on-Chip fabric.

Scale-up networking takes a practical approach as well. Instead of relying on expensive proprietary fabrics, Microsoft designed a two-tier network based on standard Ethernet with a custom transport protocol. Within each server tray, four Maia chips connect directly without external switches, exposing 2.8 TB/s of bidirectional scale-up bandwidth per accelerator. The system scales up to clusters of 6,144 chips.

Cooling is handled by Microsoft's second-generation closed-loop liquid cooling heat exchanger units. Thanks to extensive pre-silicon hardware-software co-design, Microsoft says production models were running on actual silicon within days of arrival, cutting datacenter deployment timelines by more than half.

Maia 200 is already running live in Azure's US Central region near Des Moines, Iowa, with Phoenix, Arizona scheduled as the next deployment zone. It will serve flagship models including OpenAI's GPT-5.2 across Microsoft 365 Copilot and Azure Foundry, while Microsoft's Superintelligence team uses it internally for synthetic data pipelines and reinforcement learning. Developers can now sign up for the Maia SDK preview, which includes Triton compiler integration, PyTorch support, and a local simulator.

Why it matters

We are now in the execution phase of generative AI, where operational efficiency matters just as much as benchmark scores. Maia 200 is Microsoft's declaration that it will not let external chip margins dictate the unit economics of its AI products.

For enterprise customers and developers building on Azure, this chip is all about cost and throughput. You might not care which accelerator serves your Copilot prompt today, but a 30% improvement in performance per dollar gives Microsoft room to maintain aggressive pricing on frontier models like GPT-5.2 without burning through cash. For developers working with custom pipelines, native Triton and PyTorch support means the switching cost from Nvidia-based CUDA workflows looks surprisingly manageable.

Strategically, Microsoft's decision to narrow its focus strictly to inference on TSMC's 3nm process is sharp. Training monolithic frontier models still belongs to Nvidia's top-end GPUs for now, but inference accounts for the vast majority of ongoing infrastructure spend as Copilot hits mainstream scale. By bypassing proprietary fabrics in favor of standard Ethernet, Microsoft keeps system-level TCO low.

The real test will be software stability under messy real-world traffic. Custom drivers, compilers, and transport layers are notoriously tricky to harden at scale. But if Maia 200 holds up reliably in Iowa and Arizona, Microsoft will have built a formidable defense against hardware margin squeezes across the entire cloud industry.