[论文解读] Knowledge Matters: Radiology Report Generation with General and Specific Knowledge

Shuxin Yang, Xian Wu+3 more|arXiv (Cornell University)|Dec 30, 2021

Multimodal Machine Learning Applications被引用 5

一句话总结

本文提出了一种知识增强型框架，用于胸部X光放射科报告生成，通过知识增强的多头注意力机制，整合通用知识（与图像无关）和特定知识（与图像相关）。该方法在IU-Xray和MIMIC-CXR数据集上均优于当前最先进方法，在自然语言生成（NLG）和临床有效性指标上均表现出一致的提升，消融实验验证了两类知识的有效性。

ABSTRACT

Automatic radiology report generation is critical in clinics which can relieve experienced radiologists from the heavy workload and remind inexperienced radiologists of misdiagnosis or missed diagnose. Existing approaches mainly formulate radiology report generation as an image captioning task and adopt the encoder-decoder framework. However, in the medical domain, such pure data-driven approaches suffer from the following problems: 1) visual and textual bias problem; 2) lack of expert knowledge. In this paper, we propose a knowledge-enhanced radiology report generation approach introduces two types of medical knowledge: 1) General knowledge, which is input independent and provides the broad knowledge for report generation; 2) Specific knowledge, which is input dependent and provides the fine-grained knowledge for report generation. To fully utilize both the general and specific knowledge, we also propose a knowledge-enhanced multi-head attention mechanism. By merging the visual features of the radiology image with general knowledge and specific knowledge, the proposed model can improve the quality of generated reports. Experimental results on two publicly available datasets IU-Xray and MIMIC-CXR show that the proposed knowledge enhanced approach outperforms state-of-the-art image captioning based methods. Ablation studies also demonstrate that both general and specific knowledge can help to improve the performance of radiology report generation.

研究动机与目标

• 通过整合专家医学知识，解决数据驱动放射科报告生成中的视觉和文本偏差问题。
• 克服现有编码器-解码器模型在放射科报告生成中缺乏精确专家知识整合的问题。
• 通过引入两类知识（通用知识：广泛、与图像无关；特定知识：细致、与图像相关）来提升报告质量。
• 开发一种知识增强的多头注意力机制，将视觉特征与结构化知识融合，以实现更优的报告生成。
• 通过消融实验和临床有效性指标验证通用知识与特定知识的有效性。

提出的方法

• 该模型使用预先构建的知识图谱（RadGraph）提取通用知识，表示常见的放射学术语及其关系。
• 对于特定知识，它从数据集中检索与当前输入图像最相似的前k份放射科报告，并提取相关实体和关系。
• 提出一种知识增强的多头注意力（KEMHA）机制，用于同时关注视觉特征、通用知识嵌入和特定知识嵌入。
• KEMHA机制通过融合视觉特征与知识表征来计算注意力分数，使解码器能够关注相关的医学概念。
• 知识图谱嵌入通过RotatE、TransE或TransR学习，消融实验表明RotatE表现最佳。
• 该框架采用交叉熵损失端到端训练，并在自动指标（BLEU、CIDEr）和临床有效性（CE）指标上进行评估。

Fig. 1: A radiology examination record and corresponding knowledge graph. The figure displays three parts of information. The first part includes posterior-anterior(PA) and lateral(LAT) chest x-ray images. The second part is the corresponding report which is highlighted by different relationships. T

实验结果

研究问题

• RQ1 整合通用知识是否能提升放射科报告生成的质量和一致性？
• RQ2 从相似报告中提取的图像特定知识是否能增强对细微或罕见异常的检测能力？
• RQ3 知识增强的多头注意力机制是否能有效融合视觉与知识表征，从而实现更优的报告生成？
• RQ4 不同知识图谱嵌入模型（如RotatE、TransE）对最终报告质量有何影响？
• RQ5 通用知识与特定知识的结合在多大程度上优于仅依赖视觉特征或有限知识源的模型？

主要发现

• 所提出的模型在IU-Xray数据集上达到最先进性能，使用RotatE嵌入时，BLEU-4得分为0.178，CIDEr得分为0.382。
• 在MIMIC-CXR数据集上，该模型表现与最先进方法相当，证实了其在不同数据集上的鲁棒性。
• 消融实验表明，移除通用知识嵌入（GKE）模块会导致性能下降，证实其对报告质量的积极影响。
• 移除特定知识嵌入（SKE）模块导致性能显著下降，表明图像特定知识在捕捉细微发现方面的价值。
• 知识图谱嵌入模型的选择影响性能，RotatE优于TransE和TransR，表明更高质量的知识表征能提升报告生成效果。
• 定性结果表明，该模型生成的报告更准确、更全面，对关键放射学发现（如胸腔积液和肺不张）的覆盖更好。

Fig. 2: The proposed model architecture includes four major modules: the general knowledge embedding module embeds the general knowledge (GK) and attend the visual features by knowledge-enhanced multi-head attention; the visual feature extraction module extracts visual features and predicts the dise