[Paper Review] RareBench: Can LLMs Serve as Rare Diseases Specialists?

Xuanzhong Chen, Xiaohao Mao+4 more|arXiv (Cornell University)|Feb 9, 2024

Cancer Genomics and DiagnosticsBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper introduces RareBench, a comprehensive benchmark for evaluating large language models (LLMs) in rare disease diagnosis. By leveraging a dynamic few-shot prompting method with a curated rare disease knowledge graph, GPT-4 outperforms specialist physicians in top-1 recall (0.520 vs. 0.407) on a real-world patient dataset, demonstrating strong potential for LLMs in clinical rare disease diagnostics.

ABSTRACT

Generalist Large Language Models (LLMs), such as GPT-4, have shown considerable promise in various domains, including medical diagnosis. Rare diseases, affecting approximately 300 million people worldwide, often have unsatisfactory clinical diagnosis rates primarily due to a lack of experienced physicians and the complexity of differentiating among many rare diseases. In this context, recent news such as "ChatGPT correctly diagnosed a 4-year-old's rare disease after 17 doctors failed" underscore LLMs' potential, yet underexplored, role in clinically diagnosing rare diseases. To bridge this research gap, we introduce RareBench, a pioneering benchmark designed to systematically evaluate the capabilities of LLMs on 4 critical dimensions within the realm of rare diseases. Meanwhile, we have compiled the largest open-source dataset on rare disease patients, establishing a benchmark for future studies in this domain. To facilitate differential diagnosis of rare diseases, we develop a dynamic few-shot prompt methodology, leveraging a comprehensive rare disease knowledge graph synthesized from multiple knowledge bases, significantly enhancing LLMs' diagnostic performance. Moreover, we present an exhaustive comparative study of GPT-4's diagnostic capabilities against those of specialist physicians. Our experimental findings underscore the promising potential of integrating LLMs into the clinical diagnostic process for rare diseases. This paves the way for exciting possibilities in future advancements in this field.

Motivation & Objective

• Address the critical gap in evaluating LLMs for rare disease diagnosis, where misdiagnosis rates remain high due to low physician exposure and phenotypic complexity.
• Systematically assess LLMs across four key dimensions: diagnostic accuracy, robustness, interpretability, and generalization in rare disease contexts.
• Develop a large-scale, open-source dataset of rare disease patient cases from Peking Union Medical College Hospital (PUMCH) to support future research.
• Enhance LLM diagnostic performance through a dynamic few-shot prompting strategy using a comprehensive rare disease knowledge graph.
• Compare LLMs' diagnostic performance directly against specialist physicians to establish clinical relevance and benchmark utility.

Proposed method

• Construct a rare disease knowledge graph by integrating multiple sources, including OMIM, Orphanet, CCRD, and HPO, to represent disease-phenotype relationships.
• Apply an Information-Content-based random walk algorithm to the knowledge graph to generate diverse, contextually relevant few-shot examples for prompt engineering.
• Design a dynamic few-shot prompting framework that adapts few-shot examples based on the input patient’s phenotype profile to improve diagnostic relevance.
• Use phenotype-structured input (symptoms, signs, family history) instead of raw EHR text to reduce token count and improve model focus and performance.
• Implement a controlled evaluation on 75 real PUMCH rare disease cases, comparing GPT-4 and two other LLMs against specialist physicians with and without external assistance.
• Evaluate performance using standard metrics: top-1, top-3, and top-10 recall, and median rank, to assess diagnostic precision and ranking quality.

Figure 1. RareBench ’s overview of evaluation results.

Experimental results

Research questions

• RQ1 Can LLMs achieve higher diagnostic accuracy than specialist physicians in rare disease differential diagnosis?
• RQ2 How does the use of structured phenotype input compare to raw EHR text in improving LLM diagnostic performance?
• RQ3 To what extent does a dynamic few-shot prompting strategy, informed by a rare disease knowledge graph, enhance LLM diagnostic reasoning?
• RQ4 How do diagnostic performance metrics vary across different medical specialties, especially in departments with high phenotypic overlap like cardiology?
• RQ5 What is the comparative impact of external clinical assistance on both physicians and LLMs in rare disease diagnosis?

Key findings

• GPT-4 achieved a top-1 recall of 0.520 when using phenotype-based input, significantly outperforming specialist physicians without assistance (0.407) and with assistance (0.447).
• GPT-4’s top-3 recall reached 0.747 with phenotype input, surpassing the 0.468 recall of physicians without assistance and 0.511 with assistance.
• The use of extracted phenotypes reduced input token count and improved diagnostic performance compared to raw EHR text, which yielded a top-1 recall of 0.453 for GPT-4.
• In all five medical departments evaluated, GPT-4 outperformed specialist physicians in both top-1 and top-3 recall, with the highest performance in pediatrics and the lowest in cardiology.
• Cardiology cases showed the poorest performance across all models, likely due to symptom overlap and reliance on imaging and lab results, suggesting a need for multimodal LLMs.
• The dynamic few-shot prompting method, grounded in a knowledge graph, significantly enhanced LLMs’ diagnostic reasoning and ranking accuracy on rare disease cases.

Figure 2. RareBench is the first pioneering benchmark to evaluate LLMs as rare disease specialists on 4 distinct tasks.