[Paper Review] AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

Zhihao Fan, Jialong Tang+6 more|arXiv (Cornell University)|Feb 15, 2024

Artificial Intelligence in Healthcare and Education5 citations

TL;DR

AI Hospital creates a real-time, interactive medical diagnosis environment with multi-agent LLM-powered intern doctors, patients, examiners, and a medical director, plus a Multi-View Medical Evaluation benchmark and a dispute-resolving collaboration mechanism to improve diagnostic accuracy.

ABSTRACT

Artificial intelligence has significantly advanced healthcare, particularly through large language models (LLMs) that excel in medical question answering benchmarks. However, their real-world clinical application remains limited due to the complexities of doctor-patient interactions. To address this, we introduce extbf{AI Hospital}, a multi-agent framework simulating dynamic medical interactions between \emph{Doctor} as player and NPCs including \emph{Patient}, \emph{Examiner}, \emph{Chief Physician}. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation (MVME) benchmark, utilizing high-quality Chinese medical records and NPCs to evaluate LLMs' performance in symptom collection, examination recommendations, and diagnoses. Additionally, a dispute resolution collaborative mechanism is proposed to enhance diagnostic accuracy through iterative discussions. Despite improvements, current LLMs exhibit significant performance gaps in multi-turn interactions compared to one-step approaches. Our findings highlight the need for further research to bridge these gaps and improve LLMs' clinical diagnostic capabilities. Our data, code, and experimental results are all open-sourced at \url{https://github.com/LibertFan/AI_Hospital}.

Motivation & Objective

• Motivate integration of LLMs into real-time clinical diagnosis through interactive multi-agent simulation.
• Provide a real-world-like environment with patient, examiner, intern doctor, and medical director roles to evaluate LLMs.
• Introduce MVME benchmark to assess symptom identification, examination comprehensiveness, and diagnostic quality in interactive settings.
• Demonstrate a dispute-resolution collaboration mechanism supervised by a medical director to enhance diagnostic accuracy.

Proposed method

• Construct AI Hospital with four roles (intern doctors, patients, examiners, medical director) using high-quality Chinese medical records.
• Define three-agent interaction workflow: patients describe symptoms, interns query, examiners perform tests, and the director oversees evaluation.
• Create MVME benchmark to assess symptom mastery, examination comprehensiveness, and diagnostic report quality.
• Implement a dispute-resolution collaboration where multiple interns iteratively discuss under the medical director’s guidance to reach a consensus.
• Use GPT-4 as an evaluation engine and link-based ICD-10 terminology mapping to ground diagnostic results for evaluation.
• Compare one-step (non-interactive) diagnosis against interactive collaborative setups across multiple LLMs to quantify interaction impact and collaboration gains.

Experimental results

Research questions

• RQ1 Can LLMs perform real-time, interactive clinical diagnosis in a multi-agent setting that mimics real-world consultations?
• RQ2 Does a dispute-resolution collaboration, guided by a medical director, improve diagnostic accuracy over non-collaborative approaches?
• RQ3 How do different LLMs perform on symptom gathering, test selection, and diagnostic reporting in AI Hospital?
• RQ4 What is the impact of adding more interns on diagnostic performance in collaborative diagnosis?

Key findings

• Interactive collaboration improves diagnostic accuracy over single-model, one-shot baselines.
• Collaboration with three interns yields higher diagnostic metrics than two interns, highlighting teamwork benefits.
• Dispute-resolution guided by a medical director accelerates consensus and improves evaluation outcomes.
• GPT-4-based one-step diagnosis sets an upper bound; interactive collaboration with multiple models can surpass GPT-4 alone in some metrics.
• Human evaluation shows high reliability (over 98%) in patient-examiner interactions across several models, indicating robustness of the interactive setup.
• Dispute-resolution mechanism reduces the number of rounds needed to reach consensus.