How OpenAI scaled ChatGPT storage to 1 billion users

Serving over 1 billion ChatGPT users across 40 regions is not just a GPU challenge—it is a massive data plumbing nightmare.

OpenAI’s engineering team published a deep dive into Habitat, their internal storage platform managing over 500 petabytes of data. The post details how they pushed Microsoft Azure Cosmos DB far beyond its usual limits, cycling through architectural headaches before hitting a 12x read performance boost.

The messy pendulum of multi-tenancy

Early on, OpenAI put everything into a single shared Azure Cosmos DB container. It worked until it didn't.

A single hot key from an obscure product feature would hog a physical partition, knocking down mission-critical flows like login and signup. Worse, as storage swelled, Cosmos DB split physical partitions, which actually diluted the throughput capacity assigned to each partition.

To stop the bleeding, OpenAI swung hard toward single-tenancy, telling product teams to spin up dedicated containers for each new workload. The blast radius shrank, but organizational chaos exploded.

Product engineers are not database administrators. Teams regularly launched features without pre-provisioning enough throughput, forcing infrastructure engineers to scramble in war rooms on launch days just to scale up containers.

Co-managed containers and a Rust rewrite

Realizing dedicated containers failed to scale organizationally, OpenAI partnered closely with Microsoft's Azure team to build a better multi-tenant model called co-managed containers.

They introduced smart throttling that targets the hottest keys first instead of punishing the entire physical partition. They also added request-tiering quality-of-service, ensuring low-priority bulk data jobs yield to real-time user prompts.

Paired with localized storage near users, selective replication for latency-critical data, integrated caching, and a complete proxy rewrite in Rust, the team managed to hit 12x faster reads across 500 petabytes.

Why it matters

Scaling an AI model is hard, but keeping high-concurrency state and chat histories alive for an eighth of the world's population is an entirely different beast.

For developers and infrastructure teams, this isn't a manual to copy blindly. Most companies will never touch 500 petabytes of live Cosmos DB traffic. However, the lesson on the operational tax of over-isolation is universal: letting every product team manage their own cloud resources almost always ends in SEV rooms.

We think OpenAI's decision to double down on Azure's existing primitives rather than reinventing the wheel with an in-house database was the right call. It forced Microsoft to mature Cosmos DB under extreme workloads. The trade-off, however, is clear: OpenAI's application storage is now so deeply intertwined with Azure-specific mechanics that shifting to multi-cloud down the road looks nearly impossible.