[Paper Review] An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT

Chong Ma, Zihao Wu+14 more|arXiv (Cornell University)|Apr 17, 2023

Topic ModelingComputer Science42 references40 citations

TL;DR

ImpressionGPT uses dynamic, similarity-based prompts and iterative optimization to enable ChatGPT to summarize radiology reports without model fine-tuning, achieving state-of-the-art results on MIMIC-CXR and OpenI.

ABSTRACT

The 'Impression' section of a radiology report is a critical basis for communication between radiologists and other physicians, and it is typically written by radiologists based on the 'Findings' section. However, writing numerous impressions can be laborious and error-prone for radiologists. Although recent studies have achieved promising results in automatic impression generation using large-scale medical text data for pre-training and fine-tuning pre-trained language models, such models often require substantial amounts of medical text data and have poor generalization performance. While large language models (LLMs) like ChatGPT have shown strong generalization capabilities and performance, their performance in specific domains, such as radiology, remains under-investigated and potentially limited. To address this limitation, we propose ImpressionGPT, which leverages the in-context learning capability of LLMs by constructing dynamic contexts using domain-specific, individualized data. This dynamic prompt approach enables the model to learn contextual knowledge from semantically similar examples from existing data. Additionally, we design an iterative optimization algorithm that performs automatic evaluation on the generated impression results and composes the corresponding instruction prompts to further optimize the model. The proposed ImpressionGPT model achieves state-of-the-art performance on both MIMIC-CXR and OpenI datasets without requiring additional training data or fine-tuning the LLMs. This work presents a paradigm for localizing LLMs that can be applied in a wide range of similar application scenarios, bridging the gap between general-purpose LLMs and the specific language processing needs of various domains.

Motivation & Objective

• Motivate efficient radiology report summarization and reduce radiologists’ workload by leveraging LLMs in-domain.
• Develop a dynamic, domain-specific prompting strategy that uses semantically similar existing reports to contextualize ChatGPT.
• Introduce an iterative optimization loop to refine ChatGPT outputs via automatic evaluation and prompt updates.
• Evaluate ImpressionGPT on public radiology datasets (MIMIC-CXR and OpenI) without additional LLM fine-tuning.
• Provide a generalizable paradigm for adapting generic LLMs to domain-specific tasks using limited domain data.

Proposed method

• Construct dynamic prompts by performing similarity search over a domain-specific corpus using CheXpert-derived disease labels to select semantically similar reports.
• Build a three-part dynamic prompt: task description, dynamic samples (Q&A style with similar findings and impressions), and final query.
• Apply an iterative optimization algorithm that evaluates ChatGPT outputs with Rouge-1 against impressions of similar references and updates the prompt with good/bad examples.
• Use Rouge-1 as the evaluation metric to guide iterative prompt refinement and select the best generated impression.
• Demonstrate that the approach achieves state-of-the-art performance on MIMIC-CXR and OpenI without fine-tuning the LLMs, with only a small amount of domain data used for prompting.
• Provide open-source code availability on GitHub.

Experimental results

Research questions

• RQ1 Can dynamic prompts built from semantically similar domain-specific examples improve ChatGPT's radiology impression generation without model fine-tuning?
• RQ2 Does an iterative prompt optimization loop lead to continual improvements in generated impressions compared with a static dynamic prompt?
• RQ3 How well does ImpressionGPT generalize to different radiology corpora (MIMIC-CXR and OpenI) with limited domain data?
• RQ4 What role does automatic evaluation (Rouge-1) play in guiding prompt refinement and achieving higher-quality impressions?

Key findings

• ImpressionGPT achieves substantial performance gains over existing radiology report summarization methods on two public datasets.
• The approach works with a small amount of domain data (5–20 samples) for prompting without fine-tuning the LLM.
• Dynamic context built from semantically similar reports enhances the model's ability to generate relevant impressions.
• An iterative prompt optimization loop using good/bad response feedback further improves generation quality over time.
• The framework is presented as a general paradigm for adapting large language models to domain-specific tasks with limited data, via in-context learning and iterative prompting.
• Code implementing ImpressionGPT is made available on GitHub.