[Paper Review] Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation

Fenglin Liu, Xian Wu+3 more|arXiv (Cornell University)|Jun 13, 2021

Multimodal Machine Learning Applications43 references20 citations

TL;DR

This paper proposes PPKED, a novel framework for radiology report generation that mitigates visual and textual data biases by exploring posterior knowledge (abnormal regions via disease topic tags) and prior knowledge (medical knowledge graph and prior reports), then distilling them through a multi-domain distillation module. The method outperforms prior state-of-the-art models on MIMIC-CXR and IU-Xray datasets by generating more accurate, balanced, and detailed reports with improved abnormality detection.

ABSTRACT

Automatically generating radiology reports can improve current clinical practice in diagnostic radiology. On one hand, it can relieve radiologists from the heavy burden of report writing; On the other hand, it can remind radiologists of abnormalities and avoid the misdiagnosis and missed diagnosis. Yet, this task remains a challenging job for data-driven neural networks, due to the serious visual and textual data biases. To this end, we propose a Posterior-and-Prior Knowledge Exploring-and-Distilling approach (PPKED) to imitate the working patterns of radiologists, who will first examine the abnormal regions and assign the disease topic tags to the abnormal regions, and then rely on the years of prior medical knowledge and prior working experience accumulations to write reports. Thus, the PPKED includes three modules: Posterior Knowledge Explorer (PoKE), Prior Knowledge Explorer (PrKE) and Multi-domain Knowledge Distiller (MKD). In detail, PoKE explores the posterior knowledge, which provides explicit abnormal visual regions to alleviate visual data bias; PrKE explores the prior knowledge from the prior medical knowledge graph (medical knowledge) and prior radiology reports (working experience) to alleviate textual data bias. The explored knowledge is distilled by the MKD to generate the final reports. Evaluated on MIMIC-CXR and IU-Xray datasets, our method is able to outperform previous state-of-the-art models on these two datasets.

Motivation & Objective

• To address the severe visual and textual data biases in radiology report generation, where normal findings dominate and rare abnormalities are underrepresented.
• To imitate radiologists' clinical workflow by first identifying abnormal regions and then leveraging prior medical knowledge and experience.
• To develop a framework that effectively explores and distills both posterior (image-based) and prior (knowledge-based) knowledge for improved report generation.
• To generate more accurate, diverse, and clinically relevant reports that better capture rare but critical abnormalities.

Proposed method

• The Posterior Knowledge Explorer (PoKE) uses disease topic tags to localize and attend to abnormal regions in the radiograph, reducing visual data bias.
• The Prior Knowledge Explorer (PrKE) encodes prior knowledge from a pre-constructed medical knowledge graph and retrieved prior radiology reports to reduce textual data bias.
• The Multi-domain Knowledge Distiller (MKD) adaptively fuses posterior and prior knowledge using an Adaptive Distilling Attention (ADA) mechanism that learns dynamic attention weights for each generated sentence.
• The ADA mechanism computes attention weights λ₁ and λ₂ to balance contributions from posterior and prior knowledge, with higher weights assigned to abnormality-related content.
• The framework employs a Transformer-based decoder to model long-range dependencies in report generation, improving coherence and fluency.
• Knowledge distillation is applied across multiple domains—visual, textual, and structural—ensuring alignment between generated reports and clinical findings.

Figure 1: Two examples of ground truth reports and reports generated by HRNN [ 19 ] and our method. The Red colored text indicates the abnormalities in reports. The Blue colored text stands for the similar sentences used to describe the normalities in ground truth reports. There are notable data bia

Experimental results

Research questions

• RQ1 Can posterior knowledge exploration via disease topic tags improve detection of rare and diverse abnormal regions in radiology images?
• RQ2 Can prior knowledge from medical knowledge graphs and historical reports reduce textual data bias in report generation?
• RQ3 Does adaptive knowledge distillation between posterior and prior knowledge lead to more accurate and balanced reports?
• RQ4 To what extent does the proposed PPKED framework outperform existing state-of-the-art models in radiology report generation?

Key findings

• The PPKED framework outperforms previous state-of-the-art models on both the MIMIC-CXR and IU-Xray datasets, demonstrating superior performance in report generation.
• The Adaptive Distilling Attention (ADA) mechanism successfully learns to assign higher attention weights to abnormality-related knowledge, with λ₁ and λ₂ values being significantly higher for abnormal sentences than for normal ones.
• The model generates reports with better alignment to ground truth, including accurate descriptions of rare abnormalities such as cardiomegaly, pleural effusion, and atelectasis, as shown in qualitative examples.
• The use of a Transformer-based decoder in the MKD module leads to better performance than an LSTM-based decoder, confirming its effectiveness in modeling long, coherent sequences.
• The distillation process effectively leverages prior working experience (retrieved reports) and prior medical knowledge (knowledge graph), with λ₂ being higher for normality sentences due to richer prior textual patterns.
• The AUC score for the 'Normal' category is the highest among all compared models, indicating strong performance in generating accurate normality descriptions.

Figure 2: Illustration of our proposed Posterior-and-Prior Knowledge Exploring-and-Distilling (PPKED) approach, which includes Posterior Knowledge Explorer (PoKE), Prior Knowledge Explorer (PrKE) and Multi-domain Knowledge Distiller (MKD). Specifically, PoKE explores the posterior knowledge by extra