Microsoft's AI orchestrator crushed human doctors on complex diagnoses

Facing a mysterious medical symptom often means enduring an agonizing loop of doctor visits, endless blood draws, and skyrocketing medical bills.

Microsoft AI wants to change that. The company just revealed a new diagnostic platform called the Microsoft AI Diagnostic Orchestrator, or MAI-DxO. Tested on 304 of the most grueling real-world medical cases from the New England Journal of Medicine, the system achieved an 85.5% diagnostic accuracy rate, more than four times higher than experienced human doctors.

Microsoft AI CEO Mustafa Suleyman did not hold back, calling the breakthrough a big step towards medical superintelligence.

Moving past multiple-choice exams

For years, AI models have been acing medical board exams like the USMLE with near-perfect scores. But multiple-choice tests mostly reward memorization rather than actual clinical reasoning.

Real clinical practice is iterative. A doctor hears a symptom, orders a targeted lab test, reviews the results, and updates their hypothesis before ordering the next scan. To measure this, Microsoft built SD Bench, an interactive benchmark using real Massachusetts General Hospital case proceedings.

Researchers pitted the AI against 21 practicing physicians from the US and UK with 5 to 20 years of clinical experience. Working without colleagues or reference tools, the human doctors managed an average accuracy rate of just 20% on these ultra-rare, complex cases.

A virtual panel that cuts costs

Instead of relying on a single foundation model, MAI-DxO functions as a virtual panel of specialists. Different agents collaborate to formulate hypotheses, pick targeted diagnostic tests, audit the reasoning chain, and enforce cost constraints.

When paired with OpenAI’s o3 model, MAI-DxO solved 85.5% of cases. Even better, it was significantly cheaper. The AI spent an average of $2,397 on diagnostic tests per case, compared to $2,963 spent by human physicians.

The orchestrator is model-agnostic, improving diagnostic performance across models from OpenAI, Anthropic, Google, Meta, xAI, and DeepSeek.

Why it matters

For patients and front-line doctors, a tool like this could eliminate the diagnostic wild goose chases that plague rare diseases. General practitioners cannot know every rare presentation, but a model-agnostic orchestrator brings both generalist breadth and specialist depth right to the exam room.

For the healthcare system, testing costs are a massive pain point. Healthcare spending approaches 20% of US GDP, and studies estimate up to a quarter of that is wasted on unnecessary or unhelpful testing. Proving that an AI can be both more accurate and cheaper directly addresses healthcare's biggest financial bottleneck.

We think Microsoft's decision to bake real-world diagnostic costs directly into the model's incentive structure is genuinely clever. AI agents left unchecked tends to order every test under the sun, so financial constraints act as a vital safety rail.

That said, high marks on rare NEJM cases do not automatically guarantee great performance on everyday clinic complaints like back pain or stomach bugs. Since MAI-DxO remains a research demonstration pending peer review, real-world deployment will still face rigorous clinical trials and heavy regulatory hurdles.