GPT-4 takes on radiology reports, sometimes beating doctors

When people think of AI in healthcare, they usually picture an algorithm scanning an X-ray for tumors. But in reality, radiologists spend a massive chunk of their day doing something much less glamorous: typing and summarizing reports.

At the EMNLP 2023 conference, Microsoft Research unveiled a new paper titled Exploring the Boundaries of GPT-4 in Radiology. The team partnered with Nuance, the Microsoft-owned company behind PowerScribe, a reporting tool used by more than 80 percent of US radiologists.

Together with a board-certified radiologist, they built an evaluation framework to see how well GPT-4 handles natural language tasks like disease classification, structured reporting, and findings summarization.

Outperforming humans at medical paperwork

Using techniques like zero-shot, few-shot, and chain-of-thought prompting, the researchers pitted GPT-4 against GPT-3.5 and specialized, state-of-the-art radiology models.

The results were eye-opening. On certain tasks, GPT-4 set a new state-of-the-art benchmark, beating prior models by roughly 10 percentage points in absolute improvement.

Even more surprising was the report summarization. GPT-4 generated findings summaries that were not only comparable to those written by experienced radiologists, but in some instances, evaluators actually preferred the AI's version over the human expert's.

The model also proved capable of turning unstructured, messy notes into clean, standardized formats. That makes records much easier for other doctors to read and far more useful for clinical trials relying on real-world data. Microsoft noted that the model can also translate dense medical jargon into empathetic, plain English for patients.

That said, Microsoft stressed that GPT-4 is meant to assist under strict human oversight, not replace doctors, and will require extensive clinical trials through initiatives like Project MAIRA before seeing real-world clinical deployment.

Why it matters

We think Microsoft is playing the long game smartly here. Instead of trying to replace the radiologist's core skill of reading images, which carries massive regulatory and liability hurdles, it is targeting the administrative bottleneck that doctors actually hate.

Because Nuance already owns over 80 percent of the US radiology reporting market, Microsoft does not need to convince hospitals to buy a whole new platform. They can simply weave GPT-4 capabilities directly into the daily workflow radiologists already use.

However, the margin for error in radiology is virtually zero. While an AI-generated summary might read beautifully, even a minor hallucination or omitted finding could lead to a missed diagnosis. For now, it works best as an ultra-capable drafting assistant, but the burden of proof and legal liability will remain squarely on the human holding the credentials.