[论文解读] PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation

Haibo Jin, Haoxuan Che+2 more|arXiv (Cornell University)|Aug 24, 2023

Topic ModelingComputer Science被引用 3

一句话总结

PromptMRG 提出了一种以诊断为导向的提示框架，用于医学报告生成，通过将疾病分类分支与基于标记的提示相结合，引导文本生成，从而提升诊断准确性。该方法通过结合跨模态特征增强与自适应疾病平衡学习，解决了类别不平衡问题，在两个基准数据集上实现了最先进的临床疗效表现。

ABSTRACT

Automatic medical report generation (MRG) is of great research value as it has the potential to relieve radiologists from the heavy burden of report writing. Despite recent advancements, accurate MRG remains challenging due to the need for precise clinical understanding and disease identification. Moreover, the imbalanced distribution of diseases makes the challenge even more pronounced, as rare diseases are underrepresented in training data, making their diagnostic performance unreliable. To address these challenges, we propose diagnosis-driven prompts for medical report generation (PromptMRG), a novel framework that aims to improve the diagnostic accuracy of MRG with the guidance of diagnosis-aware prompts. Specifically, PromptMRG is based on encoder-decoder architecture with an extra disease classification branch. When generating reports, the diagnostic results from the classification branch are converted into token prompts to explicitly guide the generation process. To further improve the diagnostic accuracy, we design cross-modal feature enhancement, which retrieves similar reports from the database to assist the diagnosis of a query image by leveraging the knowledge from a pre-trained CLIP. Moreover, the disease imbalanced issue is addressed by applying an adaptive logit-adjusted loss to the classification branch based on the individual learning status of each disease, which overcomes the barrier of text decoder's inability to manipulate disease distributions. Experiments on two MRG benchmarks show the effectiveness of the proposed method, where it obtains state-of-the-art clinical efficacy performance on both datasets. The code is available at https://github.com/jhb86253817/PromptMRG.

研究动机与目标

• 为解决现有医学报告生成（MRG）模型中诊断准确性较差的问题，这些模型尽管语言流畅，却常无法生成临床正确的报告。
• 提升在罕见疾病上的表现，这些疾病在当前MRG系统中因数据不平衡和诊断可靠性低而面临挑战。
• 通过使用专用分类分支生成的诊断预测，明确引导报告生成，弥合MRG模型与临床需求之间的差距。
• 通过引入自适应损失机制，克服文本解码器无法控制疾病分布的问题，实现类别不平衡学习。

提出的方法

• 采用双分支编码器-解码器架构，增加一个疾病分类分支，为每张输入图像生成诊断预测。
• 诊断驱动提示（DDP）将分类结果转换为标记提示，并将其注入解码器，以显式诊断上下文引导报告生成。
• 跨模态特征增强（CFE）利用预训练的CLIP模型，从数据库中检索与查询图像语义和视觉相似的图像-报告对，通过动态聚合增强查询图像的特征。
• 自适应疾病平衡学习（SDL）在分类分支中应用自适应logit调整损失，根据每种疾病的个体学习状态动态调整优化目标。
• 该框架端到端整合上述组件，使解码器能够生成既语言流畅又诊断准确的报告。

实验结果

研究问题

• RQ1 诊断驱动提示能否提升医学报告生成模型的临床疗效？
• RQ2 利用CLIP进行的跨模态特征增强如何提升疾病分类和报告生成的准确性？
• RQ3 自适应损失训练能否有效缓解在类别不平衡的MRG数据集中罕见疾病性能下降的问题？
• RQ4 将诊断分类与报告生成相结合，是否能比标准MRG方法更准确地对齐真实临床发现？

主要发现

• PromptMRG 在 MIMIC-CXR 和 NIH ChestX-ray 两个数据集上均实现了最先进的临床疗效（CE）表现，优于现有SOTA方法。
• 使用标记提示使F1得分相比基线绝对提升了6.4%，显著优于文本提示和特征提示。
• 在罕见疾病上，SDL模块使F1得分平均提升8%，部分疾病提升超过12%。
• 最罕见疾病的F1得分显著提高，尽管平均F1得分仅提升1.2%，表明标准指标可能低估了罕见疾病的表现。
• CFE模块通过利用CLIP检索语义和视觉上相似的报告，提升了诊断准确性，增强了分类任务的特征表示。
• 该方法表明，显式诊断提示至关重要，因为特征提示和嵌入提示的表现均劣于标记提示，凸显了结构化诊断输入的重要性。