[Paper Review] MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents

Yixing Jiang, Kameron Collin Black+5 more|ArXiv.org|Jan 24, 2025

Radiomics and Machine Learning in Medical Imaging3 citations

TL;DR

MedAgentBench provides a 100-task, 100-patient synthetic EHR benchmark with a FHIR-compliant interactive environment to evaluate medical LLM agents. GPT-4o leads overall performance among tested models, but reliability remains insufficient for clinical deployment.

ABSTRACT

Recent large language models (LLMs) have demonstrated significant advancements, particularly in their ability to serve as agents thereby surpassing their traditional role as chatbots. These agents can leverage their planning and tool utilization capabilities to address tasks specified at a high level. However, a standardized dataset to benchmark the agent capabilities of LLMs in medical applications is currently lacking, making the evaluation of LLMs on complex tasks in interactive healthcare environments challenging. To address this gap, we introduce MedAgentBench, a broad evaluation suite designed to assess the agent capabilities of large language models within medical records contexts. MedAgentBench encompasses 300 patient-specific clinically-derived tasks from 10 categories written by human physicians, realistic profiles of 100 patients with over 700,000 data elements, a FHIR-compliant interactive environment, and an accompanying codebase. The environment uses the standard APIs and communication infrastructure used in modern EMR systems, so it can be easily migrated into live EMR systems. MedAgentBench presents an unsaturated agent-oriented benchmark that current state-of-the-art LLMs exhibit some ability to succeed at. The best model (Claude 3.5 Sonnet v2) achieves a success rate of 69.67%. However, there is still substantial space for improvement which gives the community a next direction to optimize. Furthermore, there is significant variation in performance across task categories. MedAgentBench establishes this and is publicly available at https://github.com/stanfordmlgroup/MedAgentBench , offering a valuable framework for model developers to track progress and drive continuous improvements in the agent capabilities of large language models within the medical domain.

Motivation & Objective

• Motivate and enable standardized evaluation of LLMs as autonomous medical agents beyond QA chatbots.
• Provide a realistic, interactive EHR benchmark that mirrors clinical workflows and data diversity.
• Offer a reusable codebase and FHIR-based environment to migrate benchmarks into live EMR settings.

Proposed method

• Create 100 clinically derived tasks across 10 categories written by licensed clinicians.
• Generate 100 de-identified patient profiles with over 700,000 data elements to simulate real-world EMR data.
• Build a FHIR-compliant interactive environment (HAPI FHIR JPA) and a Dockerized deployment for easy setup.
• Evaluate multiple SOTA LLMs as agents using a simple agent system with nine FHIR functions and a fixed interaction budget (8 rounds).
• Define task success as the main metric, with manual references for query tasks and rule-based sanity checks for action tasks.

Figure 1: Schematic diagram of MedAgentBench architecture.

Experimental results

Research questions

• RQ1 How capable are current state-of-the-art LLMs at performing agent-style tasks in a medical EHR setting?
• RQ2 How does performance vary between information-retrieval (query) tasks and record-modification (action) tasks?
• RQ3 What are the failure modes and common error patterns of prominent LLMs when acting as medical agents?
• RQ4 To what extent can a FHIR-based interactive environment support migration to real EMR systems for benchmarking?
• RQ5 What gaps remain for reliable, clinically deployable medical AI agents?

Key findings

• GPT-4o achieves the highest overall success rate among the tested models (72% overall SR; 76% query SR; 68% action SR per Table 3).
• Most models perform better on query-based tasks than on action-based tasks, indicating information retrieval is comparatively easier for agents.
• There is substantial variation in performance across task categories and across models, with open-weight models generally underperforming closed-weight API models in this benchmark.
• Gemini 2.0 Flash shows notably low success rates and frequently outputs invalid actions or formats (e.g., tool_code blocks or non-numeric outputs).
• The benchmark reveals a non-trivial potential for medical agents but highlights the current reliability gap for safe, deployment-ready autonomous EHR automation.

Figure 2: Screenshot of frontend of the FHIR-compliant interactive environment.