[Paper Review] Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards

Jean-Benoit Delbrouck, Pierre Chambon+4 more|arXiv (Cornell University)|Oct 21, 2022

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper proposes a reinforcement learning framework that improves factual correctness in radiology report generation by leveraging semantic rewards derived from the RadGraph dataset, which annotates entities and relations in chest X-ray reports. By optimizing these graph-based rewards, the model achieves up to 25.3% relative improvement on factual correctness metrics over prior methods.

ABSTRACT

Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. These systems have achieved promising performance as measured by widely used NLG metrics such as BLEU and CIDEr. However, the current systems face important limitations. First, they present an increased complexity in architecture that offers only marginal improvements on NLG metrics. Secondly, these systems that achieve high performance on these metrics are not always factually complete or consistent due to both inadequate training and evaluation. Recent studies have shown the systems can be substantially improved by using new methods encouraging 1) the generation of domain entities consistent with the reference and 2) describing these entities in inferentially consistent ways. So far, these methods rely on weakly-supervised approaches (rule-based) and named entity recognition systems that are not specific to the chest X-ray domain. To overcome this limitation, we propose a new method, the RadGraph reward, to further improve the factual completeness and correctness of generated radiology reports. More precisely, we leverage the RadGraph dataset containing annotated chest X-ray reports with entities and relations between entities. On two open radiology report datasets, our system substantially improves the scores up to 14.2% and 25.3% on metrics evaluating the factual correctness and completeness of reports.

Motivation & Objective

• To address the persistent issue of factual incompleteness and inconsistency in neural radiology report generation systems.
• To overcome limitations of existing NLG metrics like BLEU and CIDEr, which fail to capture factual accuracy.
• To develop a domain-specific, factually grounded reward mechanism tailored to chest X-ray reports using radiologist-annotated semantic graphs.
• To improve model performance by directly optimizing for factual correctness and completeness using reinforcement learning.
• To demonstrate that semantic rewards from RadGraph outperform weakly-supervised or generic biomedical NER-based methods in factual evaluation.

Proposed method

• The authors design a new reward function, the RadGraph reward, based on semantic graphs of entities and relations extracted from the RadGraph dataset.
• The RadGraph model is fine-tuned on radiology reports to generate entity and relation annotations for both reference and generated reports.
• Three variants of the RadGraph reward are defined: one for entity overlap, one for relation overlap, and one combining both, measuring structural similarity between reference and generated report graphs.
• The model is trained using a hybrid objective: negative log-likelihood (NLL) loss for fluency, BERTScore for surface-level similarity, and the RadGraph reward for factual correctness.
• Reinforcement learning is applied to optimize the RadGraph reward, enabling the model to learn to generate reports that are both fluent and factually accurate.
• The method is evaluated on two open datasets: MIMIC-CXR and Open-i, using both standard NLG metrics and factually-oriented metrics.

Experimental results

Research questions

• RQ1 Can semantic rewards derived from radiologist-annotated entity and relation graphs improve factual correctness in radiology report generation?
• RQ2 How does optimizing for RadGraph-based rewards compare to optimizing for traditional NLG metrics like BLEU or CIDEr?
• RQ3 To what extent can a lightweight, fast-to-train model architecture outperform more complex architectures when trained with factual semantic rewards?
• RQ4 What is the impact of domain-specific semantic annotations on factual consistency and completeness compared to generic biomedical NER systems?
• RQ5 Can the RadGraph reward mitigate the tendency of models to generate high-precision but grammatically incoherent reports?

Key findings

• The proposed RadGraph reward method achieved a 14.2% relative improvement on factual correctness metrics on the MIMIC-CXR dataset compared to prior methods.
• On the Open-i dataset, the method achieved a 25.3% relative improvement on factual correctness metrics, demonstrating strong generalization across datasets.
• The model outperformed previous approaches that relied on weakly-supervised NLI models or generic biomedical NER systems, even when those methods were used as evaluation metrics.
• The RadGraph-based reward significantly improved entity recall (40.9% macro-averaged) and relation recall (18.7% macro-averaged), though performance varied by label type.
• The model showed a 0% recall for the OBS-U entity label, indicating a critical gap in learning certain clinical observations, and 15% of relations were incorrectly labeled.
• Despite the computational cost of RL training (7–10 hours per epoch vs. 50 minutes for NLL), the model achieved state-of-the-art factual correctness while maintaining a fast, lightweight architecture.