[Paper Review] MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making

Yubin Kim, Chanwoo Park+8 more|arXiv (Cornell University)|Apr 22, 2024

Business Process Modeling and Analysis8 citations

TL;DR

MDAgents adaptively assigns solo, MDT, or ICT collaboration among LLMs based on medical query complexity, improving accuracy on 5 of 7 multi-modal medical benchmarks.

ABSTRACT

Foundation models are becoming valuable tools in medicine. Yet despite their promise, the best way to leverage Large Language Models (LLMs) in complex medical tasks remains an open question. We introduce a novel multi-agent framework, named Medical Decision-making Agents (MDAgents) that helps address this gap by automatically assigning a collaboration structure to a team of LLMs. The assigned solo or group collaboration structure is tailored to the medical task at hand, emulating real-world medical decision-making processes adapted to tasks of varying complexities. We evaluate our framework and baseline methods using state-of-the-art LLMs across a suite of real-world medical knowledge and medical diagnosis benchmarks, including a comparison of LLMs' medical complexity classification against human physicians. MDAgents achieved the best performance in seven out of ten benchmarks on tasks requiring an understanding of medical knowledge and multi-modal reasoning, showing a significant improvement of up to 4.2% (p < 0.05) compared to previous methods' best performances. Ablation studies reveal that MDAgents effectively determines medical complexity to optimize for efficiency and accuracy across diverse medical tasks. Notably, the combination of moderator review and external medical knowledge in group collaboration resulted in an average accuracy improvement of 11.8%. Our code can be found at https://github.com/mitmedialab/MDAgents.

Motivation & Objective

• Motivate and enable adaptive collaboration among LLMs to mirror real-world medical decision-making.
• Define a Medical Complexity Check to categorize queries into low, moderate, or high complexity.
• Design recruitment and orchestration mechanisms to form solo, MDT, or ICT configurations at inference.
• Evaluate MDAgents against solo and group baselines on diverse multi-modal medical benchmarks.
• Analyze robustness, efficiency, and consensus dynamics of collaborative AI in medical decision-support.

Proposed method

• Four-stage framework: medical complexity check, expert recruitment, inference process, and final decision making.
• Moderation-based complexity classification that maps queries to low/moderate/high complexity levels.
• Dynamic recruitment of agents into solo, multidisciplinary team (MDT), or integrated care team (ICT) structures.
• Collaborative discussion with iterative feedback and a moderator for MDT scenarios; final decision via ensemble or consensus mechanisms.
• Incorporation of prompting techniques (e.g., Chain-of-Thought, Self-Consistency, Medprompt) for solo reasoning and structured collaboration for groups.
• Evaluation using seven medical benchmarks spanning text-only, image+text, and video+text modalities (MedQA, MedMCQA, PubMedQA, DDxPlus, PMC-VQA, Path-VQA, MedVidQA).

Experimental results

Research questions

• RQ1 Can an adaptive collaboration framework improve medical decision-making accuracy over solo or static group methods across diverse medical tasks?
• RQ2 How does varying the number of collaborating agents influence accuracy and API-cost efficiency in medical AI reasoning?
• RQ3 What are the effects of modality (text, image, video) on consensus dynamics and decision quality in multi-agent medical reasoning?
• RQ4 Does a moderator-driven MDT/ICT pipeline yield robust final decisions in high-complexity medical cases?
• RQ5 How robust is the framework to temperature variations and prompt strategies in different clinical scenarios?

Key findings

• Adaptive MDAgents achieves best performance in 5 of 7 medical benchmarks, including multi-modal tasks.
• The framework optimizes accuracy and efficiency by adjusting the number of collaborating agents (often 3) to balance performance and API usage.
• Consensus dynamics show convergence of agent opinions across modalities, with faster convergence in text+video settings.
• Ablation studies confirm the value of complexity-aware recruitment and iterative MDT refinement for higher-complexity cases.
• The approach demonstrates robustness to temperature changes and maintains high accuracy where solo/group baselines struggle.