[论文解读] Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias

Zhongwei Wan, Che Liu+7 more|arXiv (Cornell University)|May 31, 2023

Multimodal Machine Learning Applications被引用 30

一句话总结

Med-UniC统一英语和西班牙语的跨语言医学视觉-语言预训练，通过引入Cross-lingual Text Alignment Regularization来减少语言偏见，在多种医学影像任务上取得最先进的结果。

ABSTRACT

The scarcity of data presents a critical obstacle to the efficacy of medical visionlanguage pre-training (VLP). A potential solution lies in the combination of datasets from various language communities. Nevertheless, the main challenge stems from the complexity of integrating diverse syntax and semantics, language-specific medical terminology, and culture-specific implicit knowledge. Therefore, one crucial aspect to consider is the presence of community bias caused by different languages. This paper presents a novel framework named Unifying Cross-Lingual Medical Vision-Language Pre-Training (Med-UniC), designed to integrate multimodal medical data from the two most prevalent languages, English and Spanish. Specifically, we propose Cross-lingual Text Alignment Regularization (CTR) to explicitly unify cross-lingual semantic representations of medical reports originating from diverse language communities. CTR is optimized through latent language disentanglement, rendering our optimization objective to not depend on negative samples, thereby significantly mitigating the bias from determining positive-negative sample pairs within analogous medical reports. Furthermore, it ensures that the cross-lingual representation is not biased toward any specific language community. Med-UniC reaches superior performance across 5 medical image tasks and 10 datasets encompassing over 30 diseases, offering a versatile framework for unifying multi-modal medical data within diverse linguistic communities. The experimental outcomes highlight the presence of community bias in cross-lingual VLP. Reducing this bias enhances the performance not only in vision-language tasks but also in uni-modal visual tasks.

研究动机与目标

• 识别并量化来自不同语言在跨语言医学VLP中产生的社区偏见。
• 提出Med-UniC及Cross-lingual Text Alignment Regularization (CTR)以统一跨语言表示。
• 展示CTR与Med-UniC在多样化医学影像任务与数据集上的有效性。
• 证明减少语言偏见能提升跨模态和单模态的视觉任务。

提出的方法

• 学习英文和西班牙文胸部X光影像及配套放射科报告的语言无关表示。
• 使用三种并行对齐策略：cross-lingual vision-language alignment (CVL)、self-supervised vision alignment (SSV)、和 cross-lingual text alignment regularization (CTR)。
• 用生物医药语言模型的跨语言适配（CXR-BERT）的跨语言医学文本编码初始化，并构建双语词汇。
• 通过样本级和特征级去相关目标应用跨语言文本对齐正则化(CTR)，以最小化语言差异。
• 将总损失优化为 L = L_CVL + L_SSV + L_CTR，以学习视觉不变性、视觉-文本不变性和文本不变性。
• ] ,
• research_questions:[

实验结果

研究问题

• RQ1 Does community bias due to language affect cross-lingual medical VLP performance on vision-language and uni-modal tasks?
• RQ2 Can a negative-free cross-lingual text alignment regularization (CTR) unify cross-lingual representations and reduce language bias?
• RQ3 What is the impact of Med-UniC on zero-shot, linear classification, segmentation, and detection tasks across English and Spanish medical data?
• RQ4 How does Med-UniC compare with large vision models and language models in cross-lingual medical VLP?

主要发现

• Med-UniC identifies and mitigates language-based community bias in cross-lingual medical VLP.
• CTR unifies cross-lingual text representations and reduces language-specific clustering in the latent space.
• Med-UniC achieves state-of-the-art results across multiple vision-language tasks and datasets in English and Spanish.
• Med-UniC also improves performance on uni-modal visual tasks such as linear classification, segmentation, and object detection.
• Compared with large vision models, Med-UniC with ViT backbones matches or exceeds performance in several downstream tasks.
• CTR contributes substantial gains across both cross-lingual and uni-modal tasks as demonstrated by ablation studies.