[论文解读] EyeCLIP: A visual-language foundation model for multi-modal ophthalmic image analysis

Danli Shi, Weiyi Zhang+9 more|arXiv (Cornell University)|Sep 10, 2024

Retinal Imaging and Analysis被引用 7

一句话总结

EyeCLIP 提出了一种视觉–语言基础模型，在超过 2.77 million 张多模态眼科影像（含部分文本）的数据上训练，以利用多视角、多模态数据，覆盖广泛的眼科及全身疾病任务，在多任务和少样本/零样本能力方面达到当前技术水平之最。

ABSTRACT

Early detection of eye diseases like glaucoma, macular degeneration, and diabetic retinopathy is crucial for preventing vision loss. While artificial intelligence (AI) foundation models hold significant promise for addressing these challenges, existing ophthalmic foundation models primarily focus on a single modality, whereas diagnosing eye diseases requires multiple modalities. A critical yet often overlooked aspect is harnessing the multi-view information across various modalities for the same patient. Additionally, due to the long-tail nature of ophthalmic diseases, standard fully supervised or unsupervised learning approaches often struggle. Therefore, it is essential to integrate clinical text to capture a broader spectrum of diseases. We propose EyeCLIP, a visual-language foundation model developed using over 2.77 million multi-modal ophthalmology images with partial text data. To fully leverage the large multi-modal unlabeled and labeled data, we introduced a pretraining strategy that combines self-supervised reconstructions, multi-modal image contrastive learning, and image-text contrastive learning to learn a shared representation of multiple modalities. Through evaluation using 14 benchmark datasets, EyeCLIP can be transferred to a wide range of downstream tasks involving ocular and systemic diseases, achieving state-of-the-art performance in disease classification, visual question answering, and cross-modal retrieval. EyeCLIP represents a significant advancement over previous methods, especially showcasing few-shot, even zero-shot capabilities in real-world long-tail scenarios.

研究动机与目标

• 将多模态整合用于眼科疾病诊断，超越单一模态的局限。
• 利用大规模的多模态未标注和标注数据，构建统一的视觉–语言模型。
• 开发预训练策略，结合自监督重建、多模态图像对比学习以及图像-文本对比学习。

提出的方法

• 在超过 2.77 million 张多模态眼科影像（含部分文本）上对 EyeCLIP 进行预训练。
• 将自监督重建与多模态图像对比学习相结合。
• 纳入图像-文本对比学习以对齐视觉与文本表征。
• 在多种眼科模态间学习共享表示，以支持下游任务。
• 在 14 个基准数据集上评估，以评估对眼科和全身疾病任务的迁移。

实验结果

研究问题

• RQ1 一个视觉–语言基础模型是否能有效融合多视角、多模态眼科数据（图像与部分文本）以完成多样化的诊断任务？
• RQ2 通过自监督预训练、跨模态对比学习和图像-文本对齐，是否能提升下游眼科分类、VQA 与跨模态检索的性能，包括少样本/零样本场景？

主要发现

• EyeCLIP 在疾病分类、视觉问答和跨模态检索等方面，在 14 个基准数据集上实现了 state-of-the-art 的性能。
• 该模型在长尾、真实场景中展现了少样本与零样本能力。
• 该方法通过统一的预训练策略同时利用未标注和标注数据，提升了跨模态和疾病的泛化能力。
• EyeCLIP 对眼科以外的眼部及全身疾病任务也表现出有效的迁移能力。