[Paper Review] Autonomous Artificial Intelligence Agents for Clinical Decision Making in Oncology

Dyke Ferber, Omar S. M. El Nahhas+12 more|arXiv (Cornell University)|Apr 6, 2024

Radiomics and Machine Learning in Medical Imaging12 citations

TL;DR

The paper presents an autonomous AI agent framework that uses a large language model (GPT-4) as a reasoning engine to orchestrate specialized clinical tools for multimodal oncology decision support, validated on complex GI cancer cases with expert-focused evaluation.

ABSTRACT

Multimodal artificial intelligence (AI) systems have the potential to enhance clinical decision-making by interpreting various types of medical data. However, the effectiveness of these models across all medical fields is uncertain. Each discipline presents unique challenges that need to be addressed for optimal performance. This complexity is further increased when attempting to integrate different fields into a single model. Here, we introduce an alternative approach to multimodal medical AI that utilizes the generalist capabilities of a large language model (LLM) as a central reasoning engine. This engine autonomously coordinates and deploys a set of specialized medical AI tools. These tools include text, radiology and histopathology image interpretation, genomic data processing, web searches, and document retrieval from medical guidelines. We validate our system across a series of clinical oncology scenarios that closely resemble typical patient care workflows. We show that the system has a high capability in employing appropriate tools (97%), drawing correct conclusions (93.6%), and providing complete (94%), and helpful (89.2%) recommendations for individual patient cases while consistently referencing relevant literature (82.5%) upon instruction. This work provides evidence that LLMs can effectively plan and execute domain-specific models to retrieve or synthesize new information when used as autonomous agents. This enables them to function as specialist, patient-tailored clinical assistants. It also simplifies regulatory compliance by allowing each component tool to be individually validated and approved. We believe, that our work can serve as a proof-of-concept for more advanced LLM-agents in the medical domain.

Motivation & Objective

• Motivate the need for domain-specific, multimodal AI in oncology and address limitations of generalist models.
• Propose a modular AI agent framework that uses an LLM as a reasoning engine to orchestrate specialized tools.
• Ground the agent with a curated oncology knowledge base and rigorous documentation retrieval.
• Evaluate the agent on realistic multimodal GI oncology cases with expert human review.
• Demonstrate regulatory and maintenance advantages of modular, tool-specific validation over monolithic models.

Proposed method

• Construct an autonomous AI agent centered on GPT-4 as the reasoning core.
• Integrate specialized tools: radiology vision (GPT-4V), histopathology gene/mutation predictors, OncoKB, web search, a calculator, and medical image segmentation (MedSAM).
• Build a Retrieval-Augmented Generation (RAG) knowledge base from ~6,800 oncology documents using embeddings and cosine similarity retrieval.
• Generate multi-step plans and subqueries; retrieve relevant passages; cite sources for each assertion.
• Evaluate tool usage, answer completeness, factual accuracy, helpfulness, and citation alignment via blinded expert review across 11 synthetic cases.
• Limitations acknowledged include single-slice radiology, GPT-4V limitations, no follow-up questions, and oncological focus; propose modular future extensions.

Experimental results

Research questions

• RQ1 Can an LLM-based agent autonomously plan and execute a sequence of specialized medical tools to support oncology decision-making?
• RQ2 Does tool-enabled reasoning improve accuracy, completeness, and evidence-grounding of clinical recommendations in multimodal oncology scenarios?
• RQ3 How well can Retrieval-Augmented Generation and modular tools align model outputs with up-to-date guidelines and literature?
• RQ4 What are the regulatory and maintenance advantages of a modular, tool-specific architecture over a monolithic generalist model?

Key findings

• Agent consistently invoked tools across cases, averaging three tool uses per patient with one reported failure and one omission.
• Histopathology-based mutation and MSI status predictions achieved high accuracy in the seven cases with TCGA data included.
• GPT-4V guided clinical decisions toward accurate disease trajectory assessments despite occasional omissions or extraneous details.
• Model completeness reached 94% across 67 essential statements evaluated by medical experts.
• Overall factual accuracy of model assertions stood at 93.6%, with 4.3% incorrect and 2.1% potentially harmful responses.
• Citations aligned to sources accounted for 82.5% of references, with 15.2% unrelated and 2.3% conflicting; hallucinations were limited.