[论文解读] Interactive and Explainable Region-guided Radiology Report Generation

Tim Tanida, Philip C. Müller+2 more|arXiv (Cornell University)|Apr 17, 2023

Multimodal Machine Learning Applications参考文献 57被引用 10

一句话总结

RGRG 通过对象检测检测解剖区域，然后生成区域特定句子，形成完整、可交互的放射科报告，并具有增强的可解释性。

ABSTRACT

The automatic generation of radiology reports has the potential to assist radiologists in the time-consuming task of report writing. Existing methods generate the full report from image-level features, failing to explicitly focus on anatomical regions in the image. We propose a simple yet effective region-guided report generation model that detects anatomical regions and then describes individual, salient regions to form the final report. While previous methods generate reports without the possibility of human intervention and with limited explainability, our method opens up novel clinical use cases through additional interactive capabilities and introduces a high degree of transparency and explainability. Comprehensive experiments demonstrate our method's effectiveness in report generation, outperforming previous state-of-the-art models, and highlight its interactive capabilities. The code and checkpoints are available at https://github.com/ttanida/rgrg .

研究动机与目标

• Motivate automatic radiology report generation to reduce radiologist workload.
• Introduce region-guided reasoning by detecting and describing salient anatomical regions.
• Provide interactive and explainable mechanisms to select regions or draw bounding boxes for targeted descriptions.
• Demonstrate improved report quality and clinical relevance on MIMIC-CXR-derived Chest ImaGenome data.

提出的方法

• Use Faster R-CNN with ResNet-50 to detect 29 anatomical chest regions and extract region features.
• Train binary region-select and abnormality classifiers to decide which regions to describe and whether they are abnormal.
• Condition a pre-trained medical GPT-2 model on region features to generate region-specific sentences via pseudo self-attention injection.
• Concatenate region sentences and apply duplicate removal to form the final report.
• Enable anatomy-based and selection-based sentence generation for interactivity and explainability.
• Train in three stages: detector only, detector with classifiers, then end-to-end with partial language-model finetuning.

实验结果

研究问题

• RQ1 Can region-level description grounded on anatomical regions improve factual completeness and consistency of radiology reports?
• RQ2 Does region-based, interactive generation provide better explainability and radiologist control compared to image-level methods?
• RQ3 How does the model perform on standard NLG metrics and clinically relevant efficacy metrics on MIMIC-CXR-derived data?
• RQ4 Is the anatomy-based and selection-based interaction robust to deviations in bounding boxes?

主要发现

• The model achieves competitive to state-of-the-art performance on standard NLG metrics and state-of-the-art METEOR scores for full report generation.
• On clinical efficacy (CE) metrics, the approach shows substantial improvements over non-RL baselines, and is competitive with RL-optimized models.
• First to report anatomy-grounded, region-specific sentence generation with explicit visual grounding of each sentence to a bounding box.
• Anatomy-based sentence generation shows METEOR gains with higher anatomy-specificity (Anatomy-Sensitivity-Ratio ≈ 1.94 for 6 regions).
• Selection-based generation demonstrates robustness to bounding box deviations in aspect ratio and scale while remaining sensitive to position, supporting interactivity in clinical workflows.