Skip to content
IFAI

Industry

OpenAI blames massive ChatGPT outage on new telemetry service

Published Reach 39

OpenAI just published a postmortem for Wednesday’s three-hour outage across ChatGPT, Sora, and its API. It turns out the culprit wasn’t a cyberattack, but an internal telemetry rollout that overloaded Kubernetes API servers and broke DNS resolution. The company is now revamping its deployment safeguards to prevent similar lockouts.

OpenAI blames massive ChatGPT outage on new telemetry service
Image: techcrunch.com

If you couldn't get ChatGPT to respond on Wednesday afternoon, it wasn't just you.

OpenAI suffered one of the longest outages in its history, knocking out ChatGPT, the Sora video generator, and its developer API for roughly three hours. Services began failing around 3 p.m. Pacific, leaving businesses and casual users completely in the dark.

While social media immediately buzzed with rumors of cyberattacks or server melting from sudden demand, OpenAI's official postmortem revealed a far more mundane culprit: a buggy telemetry service.

A monitoring tool that choked its own network

According to OpenAI, the breakdown was triggered by a new telemetry service deployed to monitor Kubernetes, the open-source software used to manage its containerized app infrastructure.

Because telemetry tools touch nearly everything under the hood, a misconfiguration in the service unintentionally sparked massive, resource-heavy API operations. OpenAI's Kubernetes API servers were instantly overwhelmed, crashing the control planes across most of its largest clusters.

From there, things quickly snowballed. The crash crippled the underlying DNS resolution that OpenAI's various products rely on to locate resources.

To make matters worse, existing DNS caching hid the real damage in the initial moments. Engineers didn't see the full scope right away, allowing the problematic deployment to keep rolling out across more systems before anyone realized what was happening.

The lockout effect

OpenAI noted that its internal alarms actually caught the anomaly a few minutes before public users felt the impact. The problem wasn't spotting the fire; it was reaching the extinguisher.

Because the Kubernetes servers were completely bogged down, technicians faced a "locked-out effect" that made remediation agonizingly slow. They simply couldn't access the core control planes to push an immediate fix.

OpenAI called the incident a rare confluence of multiple systems failing all at once, noting that its pre-deployment tests completely missed the risk. To prevent a repeat, the company promised stricter phased rollouts for backend updates and guaranteed backdoors for engineers to access Kubernetes servers even during catastrophic overloads.

Why it matters

This three-hour freeze is a rude wake-up call for every startup and enterprise building on OpenAI's API.

For developers, relying on a single frontier model provider without an automatic failover is simply bad practice today. When an outage hits, end users blame your application, not OpenAI's Kubernetes cluster. Setting up routing fallbacks to alternative models isn't optional anymore—it is table stakes.

For the AI industry, this outage highlights a growing paradox: while front-end models are advancing rapidly, back-end infrastructure is creaking under massive complexity. A misconfigured internal monitoring tool taking down global access shows just how fragile hyper-scaled stacks can be.

We think OpenAI's detailed postmortem is commendable for its honesty, but the incident proves that engineering guardrails have lagged behind product expansion. As multi-modal workloads like Sora ramp up, OpenAI will need to treat basic systems reliability with the same urgency it devotes to training new models.

Sources (1)

  1. [1]
    techcrunch.com(opens in a new tab)Originaltechcrunch.com

Best of AI, daily