Quick Facts

• AI model achieved 78.3% diagnostic accuracy compared to 55.3% and 50% for two expert physicians in Harvard study
• System identified serious conditions like flesh-eating infections 12-24 hours before human doctors became suspicious
• Researchers emphasize AI should assist, not replace, doctors in patient care

An artificial intelligence model developed by OpenAI outperformed emergency room doctors at diagnosing patients in a real-world test, according to a Harvard Medical School study published Thursday in Science journal.

The AI system achieved 78.3% diagnostic accuracy on medical cases from Beth Israel Deaconess Medical Center in Boston. Two expert physicians scored 55.3% and 50% on the same cases.

In one dramatic example, the AI correctly suspected a flesh-eating infection in an immunosuppressed transplant patient who presented with routine respiratory symptoms. The model flagged this dangerous condition 12 to 24 hours before human physicians would have identified it, said Dr. Adam Rodman, a lead author and physician at Beth Israel.

The researchers tested the AI on 76 actual emergency department cases and 143 cases from New England Journal of Medicine clinical conferences. The system provided accurate or very close diagnoses in 88.6% of cases, compared to 72.9% for the previous GPT-4 model.

“I don’t think our findings mean that AI replaces doctors,” said Arjun Manrai, lead author and director of an AI lab at Harvard Medical School. “I think it does mean that we’re witnessing a really profound change in technology that will reshape medicine.”

In another case, the AI correctly identified lupus as the underlying cause of a patient’s worsening pulmonary embolism when doctors suspected medication failure. The system scanned medical records and linked the patient’s autoimmune history to heart inflammation.

The study revealed significant limitations. The AI struggled with probability assessments, estimating pneumonia likelihood at 70% versus the scientific range of 25-42%. Critics noted the system often suggested expensive, impractical diagnostic tests.

The research only tested text-based diagnoses. Real physicians evaluate chest X-rays, imaging studies, EKGs and other data types the AI cannot process, Manrai noted.

“What I don’t want to happen involves AI doctor companies trying to either cut doctors out of the loop or minimize clinical supervision,” Rodman said. “I do not think that these results support that.”

Rodman envisions a “triadic care model” with doctor, patient and AI system working together. He emphasized that widespread adoption would require randomized controlled trials providing stronger evidence.

Prof. Ewen Harrison from University of Edinburgh called the study important, saying these systems “are starting to look like useful second-opinion tools for clinicians, particularly when it is important to consider a wider range of possible diagnoses.”

The AI received perfect clinical reasoning scores for 98% of cases it examined, compared to 35% for attending physicians. However, researchers stress the technology should augment rather than replace human medical judgment.

Read more: In real-world test, an AI model did better than ER doctors at diagnosing patients