[论文解读] Cross-Modal Contrastive Learning for Abnormality Classification and Localization in Chest X-rays with Radiomics using a Feedback Loop.

Yan Han, Chongyan Chen+5 more|arXiv (Cornell University)|Apr 11, 2021

Radiomics and Machine Learning in Medical Imaging被引用 4

一句话总结

本文提出了一种半监督的跨模态对比学习框架，将放射组学特征与胸部X光片相结合，以提升异常分类与定位性能。通过使用Grad-CAM识别显著区域，并将提取的放射组学特征作为图像特征的正样本，该方法构建了一个反馈回路，增强了特征的鲁棒性与可解释性，在NIH胸部X光数据集上实现了最先进性能。

ABSTRACT

Building a highly accurate predictive model for these tasks usually requires a large number of manually annotated labels and pixel regions (bounding boxes) of abnormalities. However, it is expensive to acquire such annotations, especially the bounding boxes. Recently, contrastive learning has shown strong promise in leveraging unlabeled natural images to produce highly generalizable and discriminative features. However, extending its power to the medical image domain is under-explored and highly non-trivial, since medical images are much less amendable to data augmentations. In contrast, their domain knowledge, as well as multi-modality information, is often crucial. To bridge this gap, we propose an end-to-end semi-supervised cross-modal contrastive learning framework, that simultaneously performs disease classification and localization tasks. The key knob of our framework is a unique positive sampling approach tailored for the medical images, by seamlessly integrating radiomic features as an auxiliary modality. Specifically, we first apply an image encoder to classify the chest X-rays and to generate the image features. We next leverage Grad-CAM to highlight the crucial (abnormal) regions for chest X-rays (even when unannotated), from which we extract radiomic features. The radiomic features are then passed through another dedicated encoder to act as the positive sample for the image features generated from the same chest X-ray. In this way, our framework constitutes a feedback loop for image and radiomic modality features to mutually reinforce each other. Their contrasting yields cross-modality representations that are both robust and interpretable. Extensive experiments on the NIH Chest X-ray dataset demonstrate that our approach outperforms existing baselines in both classification and localization tasks.

研究动机与目标

• 为解决获取密集标注的胸部X光数据（尤其是异常区域的边界框）的高成本问题。
• 通过利用未标注数据，提升深度学习模型在医学影像中的泛化能力与可解释性。
• 弥合自然图像中对比学习与医学影像中对比学习之间的差距，后者受限于数据增强手段。
• 在图像与放射组学模态之间建立反馈回路，以相互增强特征学习。
• 在最小监督条件下实现端到端的联合分类与定位。

提出的方法

• 图像编码器处理胸部X光片，生成图像特征并执行疾病分类。
• 应用Grad-CAM突出显示X光片中的异常区域，即使无手动标注亦可实现。
• 从Grad-CAM识别的区域中提取放射组学特征，并通过专用编码器处理。
• 将放射组学特征作为同一X光片图像特征的正样本，实现跨模态对比学习。
• 建立反馈回路，使图像与放射组学特征通过对比损失相互增强，以提升表征质量。
• 通过图像与放射组学模态特征之间的对比损失，端到端训练该框架，促进判别性与可解释性表征。

实验结果

研究问题

• RQ1 放射组学特征能否在胸部X光分析的对比学习中作为有效的正样本？
• RQ2 将放射组学特征与深度图像特征结合，如何提升分类与定位性能？
• RQ3 图像与放射组学模态之间的反馈回路在多大程度上能增强模型的鲁棒性与可解释性？
• RQ4 该框架在保持高性能的同时，能否减少对昂贵边界框标注的依赖？
• RQ5 与现有医学影像领域的对比学习方法相比，该方法表现如何？

主要发现

• 所提出的框架在NIH胸部X光数据集的分类与定位任务上均实现了最先进性能。
• 将放射组学特征作为正样本显著提升了学习表征的判别能力。
• 图像与放射组学模态特征之间的反馈回路增强了特征的鲁棒性与可解释性。
• 通过利用Grad-CAM识别显著区域，该方法降低了对昂贵边界框标注的依赖。
• 跨模态对比学习框架在零样本与少样本设置下均优于现有基线方法。