ChatGPT beats doctors at medical diagnoses

Medical researchers ran an experiment to see how much AI could sharpen a doctor's clinical judgment. The results, however, were nothing short of humbling.

A small study evaluating diagnostic accuracy found that doctors armed with ChatGPT barely improved their performance compared to those working unassisted. The real plot twist came when ChatGPT took the test entirely on its own: the AI crushed the humans by 16 percentage points.

The open-book test that changed little

The study recruited 50 physicians, consisting of 26 attending physicians and 24 residents.

Every participant was tasked with diagnosing six different medical cases using the exact same clinical case histories. One group had to rely solely on their own medical training, while another group was given access to OpenAI’s ChatGPT as an assistant.

Physicians diagnosing on their own scored an average of 74%.

Doctors who had ChatGPT at their fingertips barely moved the needle, posting an average score of 76%.

Meanwhile, ChatGPT operating completely solo racked up an average score of 90%, easily beating both groups of trained clinicians.

Why humans ignored the bot

Dr. Rodman, an investigator who helped design the study, told The New York Times that he was stunned twice. First, by how little AI moved the needle for practicing doctors, and second, by how dominant the standalone model was.

So why did doctors with a 90% tool in their hands only score 76%?

According to Rodman, the primary culprit was cognitive bias: physicians simply did not listen to the AI when its suggestions contradicted their own preconceptions. Most doctors were married to their initial instincts and refused to believe a software bot knew better.

Beyond professional stubbornness, there was a technical gap. Many participating doctors did not know how to effectively prompt or interact with the chatbot to extract its full diagnostic potential.

The study authors concluded that because the AI alone outperformed both cohorts of doctors, healthcare systems urgently need both better workflow engineering and clinical workforce training to turn "AI collaboration" into actual patient outcomes.

Why it matters

This study exposes the single biggest bottleneck facing modern enterprise AI: human pride and bad workflows.

For months, the tech industry assumed that pairing a smart model with a skilled professional would automatically produce superhuman results. We think this experiment proves the exact opposite: an uncooperative expert can easily drag down a superior model to their own baseline.

For patients and developers, there are clear takeaways. ChatGPT is already remarkably good at clinical differential diagnosis when given structured case notes, making it a powerful sparring partner for complex cases. But dropping a raw chatbot onto a busy doctor's desk is clearly the wrong UX.

If health tech companies want these tools adopted in hospitals, AI cannot just spit out answers that trigger a physician's defense mechanisms. It needs to provide transparent reasoning, highlight missed edge cases, and blend seamlessly into the electronic health record. Until doctors learn how to prompt and when to listen, having frontier intelligence in the clinic will mean very little.