[Paper Review] Towards Democratization of Subspeciality Medical Expertise

Jack W. O’Sullivan, Anil Palepu+24 more|arXiv (Cornell University)|Oct 1, 2024

Biomedical Ethics and RegulationMedicine3 citations

TL;DR

This study evaluates AMIE, a large language model (LLM)-based AI system optimized for diagnostic dialogue, in augmenting general cardiologists' decision-making for complex genetic cardiomyopathies. In a blinded comparison, AMIE outperformed general cardiologists in 5 of 10 clinical evaluation domains and significantly improved cardiologists’ diagnostic quality when used as an assistive tool, suggesting LLMs can help democratize subspecialty expertise in cardiology.

ABSTRACT

The scarcity of subspecialist medical expertise, particularly in rare, complex and life-threatening diseases, poses a significant challenge for healthcare delivery. This issue is particularly acute in cardiology where timely, accurate management determines outcomes. We explored the potential of AMIE (Articulate Medical Intelligence Explorer), a large language model (LLM)-based experimental AI system optimized for diagnostic dialogue, to potentially augment and support clinical decision-making in this challenging context. We curated a real-world dataset of 204 complex cases from a subspecialist cardiology practice, including results for electrocardiograms, echocardiograms, cardiac MRI, genetic tests, and cardiopulmonary stress tests. We developed a ten-domain evaluation rubric used by subspecialists to evaluate the quality of diagnosis and clinical management plans produced by general cardiologists or AMIE, the latter enhanced with web-search and self-critique capabilities. AMIE was rated superior to general cardiologists for 5 of the 10 domains (with preference ranging from 9% to 20%), and equivalent for the rest. Access to AMIE's response improved cardiologists' overall response quality in 63.7% of cases while lowering quality in just 3.4%. Cardiologists' responses with access to AMIE were superior to cardiologist responses without access to AMIE for all 10 domains. Qualitative examinations suggest AMIE and general cardiologist could complement each other, with AMIE thorough and sensitive, while general cardiologist concise and specific. Overall, our results suggest that specialized medical LLMs have the potential to augment general cardiologists' capabilities by bridging gaps in subspecialty expertise, though further research and validation are essential for wide clinical utility.

Motivation & Objective

• Address the critical shortage of subspecialist cardiologists, especially for rare and life-threatening conditions like hypertrophic cardiomyopathy (HCM), which affects 60% of US patients undiagnosed due to lack of access.
• Evaluate whether an LLM-based AI system (AMIE) can replicate or surpass subspecialist-level diagnostic and management planning in complex cardiovascular cases.
• Investigate whether access to AMIE’s responses enhances the quality of clinical decisions made by general cardiologists.
• Develop and validate a ten-domain rubric for blinded, expert evaluation of diagnostic and management quality in complex cardiology cases.
• Contribute a novel, open-source dataset of 204 real-world cases from Stanford’s Center for Inherited Cardiovascular Disease to support future research in medical LLMs.

Proposed method

• Curated a real-world dataset of 204 complex cardiology cases from Stanford’s SCICD, including ECGs, echocardiograms, cardiac MRI, genetic tests, and cardiopulmonary stress tests.
• Developed AMIE, a fine-tuned LLM optimized for diagnostic dialogue, enhanced with web-search and self-critique capabilities to improve reasoning and evidence integration.
• Designed a ten-domain evaluation rubric used by subspecialists to assess diagnostic accuracy, differential diagnosis, risk stratification, and management planning.
• Conducted a blinded, pairwise comparison between AMIE-generated and general cardiologist-generated clinical plans, with subspecialists rating preference across domains.
• Evaluated the impact of AMIE access on general cardiologists by comparing their unassisted responses to responses made after viewing AMIE’s output.
• Conducted qualitative simulations on four patient cases to illustrate potential clinical applications of AMIE in patient communication and diagnostic reasoning.

Experimental results

Research questions

• RQ1 Can an LLM-based AI system (AMIE) produce diagnostic and management plans in complex genetic cardiomyopathies that are preferred over those of general cardiologists by subspecialists?
• RQ2 To what extent does access to AMIE’s output improve the quality of clinical decisions made by general cardiologists in subspecialty-level cases?
• RQ3 How do the strengths and limitations of AMIE compare to those of general cardiologists in terms of diagnostic thoroughness, specificity, and clinical accuracy?
• RQ4 What is the potential of specialized LLMs to bridge gaps in subspecialty expertise and support broader access to high-quality care?
• RQ5 Can a standardized, expert-validated rubric reliably benchmark AI and human performance in complex medical decision-making?

Key findings

• AMIE was rated superior to general cardiologists in 5 out of 10 evaluation domains, with preference ranging from 9% to 20%, indicating measurable improvement in diagnostic and management quality.
• In all 10 domains, general cardiologists produced higher-quality responses when they had access to AMIE’s output compared to their unassisted responses, demonstrating a consistent augmentative effect.
• AMIE improved overall response quality in 63.7% of cases when used as an assistive tool, while degrading quality in only 3.4% of cases, indicating a strong net positive impact.
• The combination of AMIE’s thorough, sensitive diagnostic reasoning and general cardiologists’ concise, specific confirmatory insights mirrors the clinical logic of a sensitive initial test followed by a specific confirmatory test.
• AMIE exhibited a higher rate of clinically significant errors compared to general cardiologists, highlighting the need for rigorous validation and safety checks before clinical deployment.
• The study contributes an open-source dataset of 204 real-world complex cardiology cases with multimodal data, enabling future research in medical LLMs and subspecialty AI.