[Paper Review] Enhancing LLMs for Impression Generation in Radiology Reports through a Multi-Agent System

Fang Zeng, Zhiliang Lyu+2 more|arXiv (Cornell University)|Dec 6, 2024

Semantic Web and Ontologies4 citations

TL;DR

This paper proposes RadCouncil, a multi-agent LLM framework that enhances radiology impression generation by integrating a Retrieval agent, Radiologist agent, and Reviewer agent. The system improves diagnostic accuracy, stylistic consistency, and clarity—demonstrating superior performance over single-agent models through iterative feedback and retrieval-augmented generation, with GPT-4 evaluation confirming significant qualitative gains.

ABSTRACT

This study introduces "RadCouncil," a multi-agent Large Language Model (LLM) framework designed to enhance the generation of impressions in radiology reports from the finding section. RadCouncil comprises three specialized agents: 1) a "Retrieval" Agent that identifies and retrieves similar reports from a vector database, 2) a "Radiologist" Agent that generates impressions based on the finding section of the given report plus the exemplar reports retrieved by the Retrieval Agent, and 3) a "Reviewer" Agent that evaluates the generated impressions and provides feedback. The performance of RadCouncil was evaluated using both quantitative metrics (BLEU, ROUGE, BERTScore) and qualitative criteria assessed by GPT-4, using chest X-ray as a case study. Experiment results show improvements in RadCouncil over the single-agent approach across multiple dimensions, including diagnostic accuracy, stylistic concordance, and clarity. This study highlights the potential of utilizing multiple interacting LLM agents, each with a dedicated task, to enhance performance in specialized medical tasks and the development of more robust and adaptable healthcare AI solutions.

Motivation & Objective

• To address the limitations of single-agent LLMs in generating accurate and consistent radiology report impressions, particularly hallucinations and stylistic inconsistencies.
• To develop a scalable, modular framework that mimics clinical radiology workflows by integrating retrieval, generation, and peer review.
• To improve diagnostic precision and stylistic alignment with radiological reporting standards using retrieval-augmented generation and multi-agent feedback.
• To evaluate the effectiveness of a multi-agent system in reducing inconsistencies, especially in temporal and status-related conclusions.
• To explore the potential of multi-agent systems for robust, reliable, and adaptable healthcare AI applications beyond radiology.

Proposed method

• RadCouncil employs three specialized LLM agents: a Retrieval agent that uses vector similarity to fetch similar historical reports from a database.
• A Radiologist agent generates impressions based on the input finding section and retrieved exemplar reports, leveraging context from prior cases.
• A Reviewer agent evaluates the generated impression for consistency with findings, focusing on detecting errors such as incorrect 'unchanged' status claims.
• The system uses iterative feedback: if inconsistencies are detected, the Radiologist agent revises the impression based on the Reviewer’s feedback.
• The framework integrates retrieval-augmented generation (RAG) to enhance domain-specific knowledge and stylistic fidelity to radiological conventions.
• Performance is evaluated using BLEU, ROUGE, BERTScore, and qualitative assessment via GPT-4, with chest X-ray reports as the test case.

Figure 1 : Diagram of the proposed RadCouncil framework, illustrating the interactions between the three agents, the report database, and the user.

Experimental results

Research questions

• RQ1 Can a multi-agent LLM framework improve the diagnostic accuracy and stylistic consistency of radiology impression generation compared to single-agent approaches?
• RQ2 How effective is the integration of retrieval-augmented generation with a dedicated review agent in reducing hallucinations and inconsistencies?
• RQ3 To what extent can a Reviewer agent detect and correct errors related to temporal status changes, such as incorrect 'unchanged' conclusions?
• RQ4 How does the multi-agent architecture, with role specialization, enhance reliability and robustness in medical text generation?
• RQ5 Can this framework be generalized to other clinical documentation tasks requiring coordination, expertise, and validation?

Key findings

• RadCouncil significantly outperformed single-agent baselines in quantitative metrics, with improvements in BLEU, ROUGE, and BERTScore, indicating better fluency and relevance.
• GPT-4 evaluation confirmed that RadCouncil-generated impressions were more clinically accurate, stylistically consistent, and clearer than those from single-agent models.
• The Reviewer agent successfully identified and corrected inconsistencies in four out of five cases involving status misstatements, particularly in 'unchanged' conclusions.
• The Reviewer agent failed to detect inconsistencies related to 'Status' when the absence of prior comparison studies was not explicitly stated, indicating a need for better temporal reasoning.
• Retrieval-augmented generation enhanced stylistic alignment with radiological reporting standards and improved diagnostic precision, though it introduced new inconsistency risks.
• The multi-agent system demonstrated the feasibility of using specialized, collaborative agents to reduce hallucinations and improve reliability in medical LLM applications.

Figure 2 : Example comparison of original impression generated by radiologist agent only vs. impression enhanced by report retriever agent.