Study explores AI’s ability to improve differential diagnosis accuracy

The development of more accessible artificial intelligence (AI) models has transformed the field of health diagnoses and medicine, with AI being used for diagnostic accuracy, personalized treatment plans, interpreting medical images, streamlining operations, supporting remote patient monitoring and much more.

Researchers from the eHealth Lab at Florida State University’s School of Information have been evaluating the application of AI as a tool to aid health care providers in making more accurate patient diagnoses. The advancement has the potential to enhance treatment methods and improve patient outcomes.

Senior author and director for FSU’s Institute for Successful Longevity Zhe He and visiting assistant professor Balu Bhasuran are among the co-authors on the multi-institutional research.

The paper, which was published in npj Digital Medicine, expands on FSU’s LabGenie project, a patient-engagement tool aimed at improving older adults’ understanding of lab test results.

The research team has been exploring the feasibility of using large language models (LLMs), a type of AI that learns from a large amount of text to answer questions accurately, to assist clinicians and improve differential diagnosis accuracy and efficiency. Differential diagnosis (DDx) is a critical step in clinical decision-making, helping health care providers distinguish between conditions with similar symptoms.

“The AI-generated differential diagnosis is very comprehensive in covering all possible diagnoses for patients,” He said. “What this study helps show is how AI can potentially be used as a tool to help practitioners make more informed decisions for their patients.”

The study involved utilizing the LLMs to generate lists of the top one, five and 10 DDx for clinicians’ evaluation. Researchers assessed the accuracy and predictive power of the LLMs and examined how incorporating lab test results impacted their diagnostic accuracy.

The study tested five LLMs—GPT-4, GPT-3.5, Llama-2-70b, Claude-2 and Mixtral-8x7B—using clinical vignettes, or narrative patient-related cases, derived from 50 case reports. Their findings reveal that lab test data significantly improves diagnostic accuracy, with GPT-4 achieving the highest performance.

Specifically, GPT-4 achieved 55% top one accuracy and 60% top 10 accuracy with lab data, with lenient accuracy reaching 80%. Lab tests, including liver function, metabolic/toxicology panels and serology/immune tests, were generally interpreted correctly by the LLMs.

“When we asked the model for the top differential diagnosis, most of these models were able to produce the patient’s exact diagnosis,” Bhasuran said. “That’s very interesting because it implies that even in rare case diseases, the model is able to predict that.”

The research aims to help address well-known areas of concern often felt in health care settings from both the provider and patient perspective. Accurate diagnosis is crucial for effective patient management, directly influencing treatment decisions and overall patient outcomes.

Reducing diagnostic errors helps streamline patient care, eliminating the need for excessive or repeated testing and ultimately lowering health care costs through reduced hospital stays and unnecessary procedures.