[Paper Review] ClinicalAgent: Clinical Trial Multi-Agent System with Large Language Model-based Reasoning

Ling Yue, Xing, Sixue+2 more|arXiv (Cornell University)|Apr 23, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

CT-Agent is a novel multi-agent system that leverages GPT-4, LEAST-TO-MOST reasoning, and ReAct to autonomously manage clinical trial tasks, including outcome prediction, failure analysis, and duration estimation. It achieves a 0.7908 PR-AUC in outcome prediction, a 0.3326 improvement over standard prompting, demonstrating enhanced actionable intelligence in clinical trial applications.

ABSTRACT

Large Language Models (LLMs) and multi-agent systems have shown impressive capabilities in natural language tasks but face challenges in clinical trial applications, primarily due to limited access to external knowledge. Recognizing the potential of advanced clinical trial tools that aggregate and predict based on the latest medical data, we propose an integrated solution to enhance their accessibility and utility. We introduce Clinical Agent System (ClinicalAgent), a clinical multi-agent system designed for clinical trial tasks, leveraging GPT-4, multi-agent architectures, LEAST-TO-MOST, and ReAct reasoning technology. This integration not only boosts LLM performance in clinical contexts but also introduces novel functionalities. The proposed method achieves competitive predictive performance in clinical trial outcome prediction (0.7908 PR-AUC), obtaining a 0.3326 improvement over the standard prompt Method. Publicly available code can be found at https://anonymous.4open.science/r/ClinicalAgent-6671.

Motivation & Objective

• To address the gap in actionable intelligence in clinical trial applications of large language models (LLMs), which often remain limited to conversational interactions.
• To develop a multi-agent framework that integrates external knowledge sources and reasoning techniques to improve clinical trial decision-making.
• To enable autonomous, explainable, and precise prediction of clinical trial outcomes, including failure reasons and duration estimates.
• To enhance LLM performance in clinical settings by combining multi-agent architectures with advanced reasoning methods like LEAST-TO-MOST and ReAct.
• To establish a benchmark for LLM-based systems in clinical trials that move beyond dialogue toward actionable, data-driven insights.

Proposed method

• The system employs a multi-agent architecture with specialized agents for enrollment, safety, and efficacy tasks, each handling subproblems in clinical trial reasoning.
• It integrates external tools such as a trained enrollment success prediction model (0.359 failure probability) and historical drug safety data (100% failure rate for Aggrenox capsules).
• Reasoning is guided by the ReAct framework, which interleaves reasoning and tool use, and LEAST-TO-MOST prompting to decompose complex queries into manageable steps.
• GPT-4 serves as the core LLM, enabling advanced natural language understanding and generation across clinical trial components.
• The system uses few-shot learning to improve performance, with agents dynamically selecting and applying relevant examples during reasoning.
• A central reasoning agent synthesizes inputs from specialized agents and external tools to produce final decisions, such as predicting trial failure with high confidence.

Experimental results

Research questions

• RQ1 Can a multi-agent system with LLM-based reasoning improve clinical trial outcome prediction beyond standard prompting techniques?
• RQ2 How does integrating external knowledge sources and reasoning frameworks enhance decision-making in clinical trial management?
• RQ3 To what extent does the use of specialized agents (e.g., enrollment, safety, efficacy) improve the accuracy and explainability of clinical trial predictions?
• RQ4 How does GPT-4 compare to GPT-3.5 in predicting clinical trial outcomes using structured reasoning and tool use?
• RQ5 What is the impact of few-shot learning on the performance of a multi-agent LLM system in clinical trial reasoning tasks?

Key findings

• CT-Agent achieved a PR-AUC of 0.7908 in clinical trial outcome prediction, representing a 0.3326 improvement over the standard prompting baseline.
• GPT-4 outperformed GPT-3.5 across all metrics, with higher ROC-AUC (0.8347 vs. 0.824) and PR-AUC (0.7908 vs. 0.6793), confirming the benefit of advanced LLMs.
• The few-shot learning variant of CT-Agent showed superior performance in identifying positive cases (higher PR-AUC), despite slightly lower accuracy and F1 scores.
• The system correctly predicted a clinical trial failure due to a 100% historical failure rate of Aggrenox capsules, demonstrating effective safety reasoning.
• The enrollment agent successfully predicted a 35.9% failure probability for enrollment, showcasing the integration of predictive models into the reasoning pipeline.
• Expert feedback and computational benchmarks confirmed that CT-Agent significantly improves efficiency and decision quality in clinical trial processes.