[Paper Review] KiUT: Knowledge-injected U-Transformer for Radiology Report Generation

Zhongzhen Huang, Xiaofan Zhang+1 more|arXiv (Cornell University)|Jun 20, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

KiUT proposes a Knowledge-injected U-Transformer for radiology report generation, integrating visual, contextual, and clinical knowledge via a U-connection encoder-decoder architecture and an Injected Knowledge Distiller. It achieves state-of-the-art performance on IU-Xray and MIMIC-CXR by leveraging a symptom graph and multi-level feature interaction, significantly improving abnormality description and report coherence.

ABSTRACT

Radiology report generation aims to automatically generate a clinically accurate and coherent paragraph from the X-ray image, which could relieve radiologists from the heavy burden of report writing. Although various image caption methods have shown remarkable performance in the natural image field, generating accurate reports for medical images requires knowledge of multiple modalities, including vision, language, and medical terminology. We propose a Knowledge-injected U-Transformer (KiUT) to learn multi-level visual representation and adaptively distill the information with contextual and clinical knowledge for word prediction. In detail, a U-connection schema between the encoder and decoder is designed to model interactions between different modalities. And a symptom graph and an injected knowledge distiller are developed to assist the report generation. Experimentally, we outperform state-of-the-art methods on two widely used benchmark datasets: IU-Xray and MIMIC-CXR. Further experimental results prove the advantages of our architecture and the complementary benefits of the injected knowledge.

Motivation & Objective

• To address the limitations of existing models in generating accurate, radiologist-like reports by integrating multi-modal knowledge.
• To reduce data biases in radiology report generation, particularly overrepresentation of normal findings and regions.
• To improve report quality by modeling complex relationships between image regions and clinical entities.
• To develop a lightweight, effective architecture that enhances cross-modal interaction between vision and language.
• To validate the effectiveness of injected knowledge distillation in the decoding phase for better clinical reasoning.

Proposed method

• A U-connection architecture is designed between the encoder and decoder to enable multi-level interaction and feature fusion across visual, contextual, and clinical modalities.
• A symptom graph is constructed from expert-annotated clinical entities and relationships, providing structured medical knowledge with homogeneous embedding space.
• An Injected Knowledge Distiller is introduced on top of the decoder to dynamically distill and integrate visual, contextual, and clinical knowledge during word prediction.
• The Region Relationship Encoder captures both intrinsic (within-region) and extrinsic (between-region) relationships among image regions to enhance abnormality detection.
• The model uses a transformer-based encoder-decoder framework with multi-head self-attention and feed-forward networks, adapted for cross-modal generation.
• Knowledge injection occurs at the decoding stage, avoiding interference with visual feature learning and ensuring alignment across heterogeneous embedding spaces.

Experimental results

Research questions

• RQ1 Can a U-connection architecture between encoder and decoder improve multi-level feature interaction in radiology report generation?
• RQ2 How does injecting clinical knowledge via a symptom graph affect the accuracy and clinical relevance of generated reports?
• RQ3 To what extent does knowledge distillation from visual, contextual, and clinical features improve report generation performance?
• RQ4 Does the proposed method outperform existing state-of-the-art models on standard radiology report benchmarks?
• RQ5 How does the model handle subtle or complex abnormalities that are often missed by standard captioning models?

Key findings

• KiUT achieves state-of-the-art performance on both IU-Xray and MIMIC-CXR datasets, outperforming previous methods in BLEU, ROUGE, and other standard metrics.
• The ablation study shows that removing both contextual and clinical features reduces the BLEU-1 score from 0.393 to 0.337, confirming their critical role in report generation.
• The U-connection structure improves performance by 0.01 BLEU-1 over the standard 1-to-1 connection and matches the performance of more complex meshed connections with fewer parameters.
• Qualitative analysis confirms that KiUT accurately describes subtle findings such as 'slight elevation of the right hemidiaphragm' and 'support devices', demonstrating improved sensitivity to fine-grained abnormalities.
• The model successfully generates contextually coherent reports that reflect clinical reasoning, such as inferring cardiac silhouette assessment difficulty due to pleural effusion.
• The Injected Knowledge Distiller enables the model to reason across symptoms, as shown in generating reports that correctly link image findings with clinical implications.