[Paper Review] RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatment

Xuanzhong Chen, Ye Jin+4 more|arXiv (Cornell University)|Dec 17, 2024

Genomics and Rare DiseasesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

RareAgents is a novel, patient-centered, multi-disciplinary LLM-based agent framework designed for rare disease diagnosis and treatment, integrating dynamic memory, tool use, and collaborative planning among specialized physician agents. It outperforms SOTA models including GPT-4o and domain-specific agents in differential diagnosis and medication recommendation using Llama-3.1-8B/70B, and introduces the MIMIC-IV-Ext-Rare dataset to advance rare disease research.

ABSTRACT

Rare diseases, despite their low individual incidence, collectively impact around 300 million people worldwide due to the vast number of diseases. The involvement of multiple organs and systems, and the shortage of specialized doctors with relevant experience, make diagnosing and treating rare diseases more challenging than common diseases. Recently, agents powered by large language models (LLMs) have demonstrated notable applications across various domains. In the medical field, some agent methods have outperformed direct prompts in question-answering tasks from medical examinations. However, current agent frameworks are not well-adapted to real-world clinical scenarios, especially those involving the complex demands of rare diseases. To bridge this gap, we introduce RareAgents, the first LLM-driven multi-disciplinary team decision-support tool designed specifically for the complex clinical context of rare diseases. RareAgents integrates advanced Multidisciplinary Team (MDT) coordination, memory mechanisms, and medical tools utilization, leveraging Llama-3.1-8B/70B as the base model. Experimental results show that RareAgents outperforms state-of-the-art domain-specific models, GPT-4o, and current agent frameworks in diagnosis and treatment for rare diseases. Furthermore, we contribute a novel rare disease dataset, MIMIC-IV-Ext-Rare, to facilitate further research in this field.

Motivation & Objective

• To address the diagnostic delay and treatment challenges in rare diseases due to symptom complexity and specialist shortages.
• To develop an autonomous, multi-disciplinary team of LLM-based agents that simulate real-world clinical workflows for rare disease patients.
• To enhance diagnostic accuracy and medication recommendation by integrating planning, memory, and medical tool utilization in a patient-centered framework.
• To create a plug-and-play, extensible architecture for diverse medical decision-making scenarios beyond rare diseases.
• To contribute a new, clinically grounded dataset, MIMIC-IV-Ext-Rare, to support future research in rare disease LLM applications.

Proposed method

• The system uses a Patient Agent to represent the patient’s clinical profile and request, initiating communication with an Attending Physician Agent.
• The Attending Physician Agent dynamically assembles a multi-disciplinary team (MDT) of specialized physician agents from a predefined pool, each with domain-specific roles.
• Each physician agent employs dynamic long-term memory to retain and retrieve relevant clinical context across interactions.
• The framework enables tool use via integration with medical databases (e.g., DrugBank) and interaction checkers (e.g., DDI-graph) for drug safety validation.
• The MDT collaborates through iterative planning and reasoning, with the Attending Physician synthesizing final recommendations.
• The system is built on Llama-3.1-8B/70B as the base LLM, enabling zero-shot reasoning and agent autonomy in complex clinical scenarios.

Figure 1: The four steps of the RareAgents framework. In Step 2, the Attending Physician Agent applies a) Planning to form an MDT and reach a consensus through iterative discussions; b) Dynamic long-term memory to retrieve and update similar cases or prior records; and c) Tools to assist in diagnosi

Experimental results

Research questions

• RQ1 Can an autonomous, multi-disciplinary team of LLM agents outperform existing agent frameworks and SOTA models in diagnosing rare diseases?
• RQ2 How does the integration of memory, planning, and medical tool use improve diagnostic and treatment accuracy in rare disease scenarios?
• RQ3 To what extent can a patient-centered, plug-and-play agent framework generalize across diverse rare disease cases with complex, multi-organ presentations?
• RQ4 How does the proposed framework reduce diagnostic odyssey duration and improve medication safety through automated reasoning and interaction checks?
• RQ5 Can a newly curated dataset, MIMIC-IV-Ext-Rare, effectively support and advance research in rare disease LLM applications?

Key findings

• RareAgents significantly outperforms state-of-the-art domain-specific models and GPT-4o in differential diagnosis for rare diseases, demonstrating superior reasoning and accuracy.
• The framework achieves higher medication recommendation accuracy than existing agent-based and SOTA models, particularly in complex, multi-drug regimens.
• The integration of dynamic memory and tool use (e.g., DrugBank, DDI-graph) leads to safer and more personalized treatment plans, reducing potential drug interactions.
• Ablation studies confirm that each component—planning, memory, and tool use—contributes meaningfully to the overall performance, with memory showing the most significant impact.
• The newly introduced MIMIC-IV-Ext-Rare dataset extends MIMIC-IV with rare disease cases, enabling future benchmarking and model development in this under-resourced domain.
• In case studies, RareAgents correctly recommended 16/21 medications in a complex oncology case, with only 4 absent from ground truth, indicating strong clinical relevance and safety.

Figure 2: Illustrative examples of our proposed RareAgents framework for diagnosing and treating rare diseases are presented. The left panel features a patient from RareBench-Public requiring differential diagnosis based on symptoms, while the right panel shows a case from MIMIC-IV-Ext-Rare involvin