[Paper Review] Almanac Copilot: Towards Autonomous Electronic Health Record Navigation

Cyril Zakka, Joseph Cho+10 more|arXiv (Cornell University)|Apr 30, 2024

Electronic Health Records SystemsHealth Professions3 citations

TL;DR

Almanac Copilot is a Level 1 autonomous agent framework designed to reduce clinician cognitive load by automating electronic health record (EHR) tasks such as information retrieval, order placement, and alert surfacing. Evaluated on the EHR-QA synthetic dataset of 300 real-patient-based queries, it achieved a 74% task success rate (221/300 tasks) with a mean score of 2.45/3, demonstrating strong potential for streamlining EHR workflows.

ABSTRACT

Clinicians spend large amounts of time on clinical documentation, and inefficiencies impact quality of care and increase clinician burnout. Despite the promise of electronic medical records (EMR), the transition from paper-based records has been negatively associated with clinician wellness, in part due to poor user experience, increased burden of documentation, and alert fatigue. In this study, we present Almanac Copilot, an autonomous agent capable of assisting clinicians with EMR-specific tasks such as information retrieval and order placement. On EHR-QA, a synthetic evaluation dataset of 300 common EHR queries based on real patient data, Almanac Copilot obtains a successful task completion rate of 74% (n = 221 tasks) with a mean score of 2.45 over 3 (95% CI:2.34-2.56). By automating routine tasks and streamlining the documentation process, our findings highlight the significant potential of autonomous agents to mitigate the cognitive load imposed on clinicians by current EMR systems.

Motivation & Objective

• To address clinician burnout and EHR-related cognitive overload caused by inefficient documentation and workflow fragmentation.
• To develop an autonomous agent that performs EHR-specific tasks such as data retrieval, order placement, and alert prioritization.
• To evaluate the agent’s performance on a realistic, synthetic EHR benchmark reflecting common clinical workflows.
• To establish a foundation for future Level 2 autonomous agents with proactive, context-aware capabilities.
• To improve EHR usability by minimizing redundant data entry and alert fatigue through intelligent automation.

Proposed method

• The system uses a dynamic API selection architecture that chooses from a predefined set of tools (e.g., FHIR APIs, calculators, browser search) based on query semantics.
• It leverages large language models (LLMs) not as standalone knowledgebases, but for grounding and orchestrating external tool use within the EHR context.
• A synthetic evaluation dataset, EHR-QA, was created using 40 template questions and an off-the-shelf LLM to generate 300 grounded, real-patient-based queries.
• Tasks were evaluated across three domains: information retrieval, data manipulation (e.g., order placement), and alert surfacing, with human-in-the-loop validation.
• The agent operates at Level 1 autonomy, where clinicians review and approve all actions before execution, ensuring safety and compliance.
• Evaluation metrics included task success rate and a 3-point scoring system (1–3) for correctness and completeness.

Experimental results

Research questions

• RQ1 Can an autonomous agent effectively perform common EHR tasks such as retrieving patient test results, placing orders, and summarizing medical history?
• RQ2 How well does the agent perform on a realistic, synthetic EHR benchmark derived from real patient data?
• RQ3 To what extent can LLM-driven agents reduce clinician workload by automating routine EHR interactions?
• RQ4 How does the system balance accuracy, safety, and usability in a clinical setting with minimal clinician oversight?
• RQ5 What are the key limitations of current LLM-based agents in handling complex, multi-step clinical reasoning in EHRs?

Key findings

• Almanac Copilot achieved a 74% task success rate (221 out of 300 tasks) on the EHR-QA benchmark, indicating strong performance on real-world EHR tasks.
• The system obtained a mean score of 2.45 out of 3 (95% CI: 2.34–2.56), demonstrating high accuracy and completeness in task execution.
• The agent successfully handled 142 information retrieval tasks, 103 data manipulation tasks (e.g., order placement), and 55 alert surfacing tasks with high fidelity.
• The use of external tools (e.g., FHIR APIs, clinical calculators) significantly improved grounding and accuracy compared to LLMs alone.
• The synthetic EHR-QA dataset effectively captures the complexity of clinical workflows, enabling reliable evaluation of autonomous EHR agents.
• The results suggest that autonomous agents can significantly reduce the cognitive and administrative burden on clinicians when properly integrated into EHR systems.