OpenAI's o1 just beat doctors at ER triage

AI just beat human doctors in the emergency room, but ER physicians aren't ready to hand over their stethoscopes quite yet.

A new study published in Science from researchers at Harvard Medical School and Beth Israel Deaconess Medical Center tested OpenAI’s reasoning models against real ER cases. Across 76 real emergency room patients, OpenAI's o1 model diagnosed conditions more accurately than two internal medicine attending physicians.

Raw medical records and a blind test

The researchers didn't curate or pre-process the data. Instead, they fed the models the exact raw electronic medical record text available at the time of each clinical decision stage, comparing o1 and GPT-4o against two attending physicians.

To keep things fair, two independent attending physicians scored the results in a double-blind review, meaning they had no idea which diagnosis came from silicon and which came from a human.

At the initial triage point, where information is scarce and urgency is highest, o1 landed an exact or very close diagnosis 67.1% of the time. The two human doctors hit the mark 55.3% and 50.0% of the time, respectively. In one dramatic case, the AI flagged a rare flesh-eating infection in an organ transplant patient roughly 12 to 24 hours before the treating human doctor caught it.

ER doctors push back on the premise

While the numbers turn heads, emergency room veterans were quick to point out a glaring flaw in the study design.

Emergency physician Kristen Panthagani noted that the trial compared the AI against internal medicine doctors, not practicing emergency medicine specialists. As she put it, an LLM might beat a dermatologist on a neurosurgery board exam, but that does not prove it belongs in a brain surgery suite.

More importantly, ER diagnosis is not a trivia contest. Panthagani explained that an ER doctor's primary mission isn't guessing the final disease on first contact; it is determining whether the patient has an immediate life-threatening emergency.

Lead study author Adam Rodman also told The Guardian that the world still lacks any formal framework for AI accountability in clinical settings, adding that patients still demand human judgment when life-or-death treatment decisions are on the line.

Why it matters

We think this study marks a clear shift from sanitized medical benchmark tests toward messy, real-world utility, but it does not mean an algorithm is running the ER tomorrow.

For doctors, this level of reasoning makes LLMs a viable second pair of eyes. Catching an obscure flesh-eating bacteria a full day early is a massive clinical win. Having a model scan raw chart notes in the background could prevent fatal diagnostic blind spots during chaotic overnight shifts.

However, the study's primary metric of guessing the ultimate diagnosis misjudges what emergency medicine actually does. ER triage is about stabilizing vital signs, ruling out acute killers, and physical examination, none of which an LLM processing text can do on its own.

Until there is clear regulatory accountability and models can process physical multi-modal inputs reliably, AI will remain a backseat co-pilot. But as an always-on clinical sounding board, it is quickly proving to be an exceptionally smart one.