[Paper Review] Toward expanding the scope of radiology report summarization to multiple anatomies and modalities

Zhihong Chen, Maya Varma+3 more|arXiv (Cornell University)|Nov 15, 2022

Topic Modeling28 references4 citations

TL;DR

This paper introduces MIMIC-RRS, an open-source dataset spanning 12 modality-anatomy pairs (including X-ray, CT, and MRI across chest, head, spine, and other regions), enabling multi-modality and multi-anatomy radiology report summarization. It demonstrates that pre-trained models like BART achieve strong performance with minimal architectural complexity, while introducing F1-RadGraph—a factual correctness metric validated by radiologists—enabling reliable evaluation across diverse anatomical and imaging contexts.

ABSTRACT

Radiology report summarization (RRS) is a growing area of research. Given the Findings section of a radiology report, the goal is to generate a summary (called an Impression section) that highlights the key observations and conclusions of the radiology study. However, RRS currently faces essential limitations.First, many prior studies conduct experiments on private datasets, preventing reproduction of results and fair comparisons across different systems and solutions. Second, most prior approaches are evaluated solely on chest X-rays. To address these limitations, we propose a dataset (MIMIC-RRS) involving three new modalities and seven new anatomies based on the MIMIC-III and MIMIC-CXR datasets. We then conduct extensive experiments to evaluate the performance of models both within and across modality-anatomy pairs in MIMIC-RRS. In addition, we evaluate their clinical efficacy via RadGraph, a factual correctness metric.

Motivation & Objective

• To address the lack of publicly available, reproducible datasets for radiology report summarization (RRS) across multiple anatomies and modalities.
• To overcome the limitation of prior studies that focus exclusively on chest X-rays and private datasets, hindering fair comparison and replication.
• To develop and validate a factual correctness metric, F1-RadGraph, for evaluating generated summaries across diverse anatomical and imaging contexts.
• To benchmark pre-trained models on a new, diverse benchmark to assess generalization across modality-anatomy pairs.
• To establish a strong, simple baseline using BART, showing that complex architectures are not necessary for high performance.

Proposed method

• Constructed MIMIC-RRS by combining MIMIC-III (79,779 reports) and MIMIC-CXR (128,003 reports), covering 11 modality-anatomy pairs, including CT, MRI, and X-ray across 7 anatomies (chest, head, neck, sinus, spine, abdomen, pelvis).
• Identified 537 alternative section headers for 'Findings' across different modality-anatomy pairs with input from a board-certified radiologist to extract relevant report sections.
• Used a pre-trained radiology-specific NLP model (RadGraph) to extract medical entities and relations from both ground-truth and generated impressions for factual evaluation.
• Introduced F1-RadGraph, a metric that computes F1-score between predicted and gold-standard named entities and relations, validated by a radiologist for correctness.
• Conducted over 400 cross-modality-anatomy evaluations across 12 pairs, benchmarking multiple pre-trained models including BART, RoBERTa, and BioBART.
• Evaluated models using standard ROUGE scores and clinical F1-RadGraph scores to assess both fluency and factual accuracy.

Experimental results

Research questions

• RQ1 Can a publicly available, multi-modality, multi-anatomy radiology report summarization dataset be constructed to enable reproducible and comparable research?
• RQ2 How do pre-trained models like BART generalize across different modality-anatomy pairs in RRS, and does architectural complexity improve performance?
• RQ3 Can a factual correctness metric like F1-RadGraph be effectively applied to diverse anatomical and imaging contexts to evaluate clinical reliability?
• RQ4 Does using radiologist-annotated entity and relation data from a pre-trained model improve factual consistency over unsupervised NLP tools?
• RQ5 Is a simple backbone like BART sufficient for strong performance, or are complex modules necessary for high-quality summarization?

Key findings

• MIMIC-RRS is a publicly released dataset of 207,782 radiology reports across 12 modality-anatomy pairs, including CT, MRI, and X-ray across 7 anatomical regions, enabling broader evaluation than prior work.
• BART-based models achieved the highest F1-RadGraph scores, demonstrating that strong performance can be achieved with a standard pre-trained model without complex architectural modifications.
• F1-RadGraph scores showed strong consistency with ROUGE scores, validating its reliability as a factual correctness metric across diverse anatomical and imaging contexts.
• The dataset includes 11 in-domain and 6 out-of-domain (OOD) modality-anatomy pairs, enabling evaluation of generalization across previously unseen combinations.
• The use of radiologist-validated entity and relation annotations in RadGraph improved factual consistency evaluation compared to unsupervised NLP tools like Stanza.
• Despite the complexity of recent models, the study shows that simpler models like BART can achieve competitive results, suggesting that architectural innovation may not be the primary driver of performance gains.