[Paper Review] EyeCLIP: A visual-language foundation model for multi-modal ophthalmic image analysis

Danli Shi, Weiyi Zhang+9 more|arXiv (Cornell University)|Sep 10, 2024

Retinal Imaging and Analysis7 citations

TL;DR

EyeCLIP proposes a visual–language foundation model trained on over 2.77 million multi-modal ophthalmology images with partial text to leverage multi-view, multi-modal data for broad ophthalmic and systemic disease tasks, achieving state-of-the-art performance across tasks and few-/zero-shot capabilities.

ABSTRACT

Early detection of eye diseases like glaucoma, macular degeneration, and diabetic retinopathy is crucial for preventing vision loss. While artificial intelligence (AI) foundation models hold significant promise for addressing these challenges, existing ophthalmic foundation models primarily focus on a single modality, whereas diagnosing eye diseases requires multiple modalities. A critical yet often overlooked aspect is harnessing the multi-view information across various modalities for the same patient. Additionally, due to the long-tail nature of ophthalmic diseases, standard fully supervised or unsupervised learning approaches often struggle. Therefore, it is essential to integrate clinical text to capture a broader spectrum of diseases. We propose EyeCLIP, a visual-language foundation model developed using over 2.77 million multi-modal ophthalmology images with partial text data. To fully leverage the large multi-modal unlabeled and labeled data, we introduced a pretraining strategy that combines self-supervised reconstructions, multi-modal image contrastive learning, and image-text contrastive learning to learn a shared representation of multiple modalities. Through evaluation using 14 benchmark datasets, EyeCLIP can be transferred to a wide range of downstream tasks involving ocular and systemic diseases, achieving state-of-the-art performance in disease classification, visual question answering, and cross-modal retrieval. EyeCLIP represents a significant advancement over previous methods, especially showcasing few-shot, even zero-shot capabilities in real-world long-tail scenarios.

Motivation & Objective

• Motivate multi-modal integration for ophthalmic disease diagnosis beyond single modalities.
• Leverage large-scale multi-modal unlabeled and labeled data with a unified visual-language model.
• Develop pretraining strategies that combine self-supervised reconstructions, multi-modal image contrastive learning, and image-text contrastive learning.

Proposed method

• Pretrain EyeCLIP on over 2.77 million multi-modal ophthalmology images with partial text data.
• Combine self-supervised reconstruction with multi-modal image contrastive learning.
• Incorporate image-text contrastive learning to align visual and textual representations.
• Learn a shared representation across multiple ophthalmic modalities to support downstream tasks.
• Evaluate on 14 benchmark datasets to assess transfer to ocular and systemic disease tasks.

Experimental results

Research questions

• RQ1 Can a visual-language foundation model effectively fuse multi-view, multi-modal ophthalmic data (images and partial text) for diverse diagnostic tasks?
• RQ2 Does pretraining with self-supervision, cross-modal contrastive learning, and image-text alignment improve performance on downstream ophthalmic classification, VQA, and cross-modal retrieval, including few-/zero-shot scenarios?

Key findings

• EyeCLIP achieves state-of-the-art performance on disease classification, visual question answering, and cross-modal retrieval across 14 benchmark datasets.
• The model demonstrates few-shot and zero-shot capabilities in long-tail, real-world scenarios.
• The approach leverages both unlabeled and labeled data through a unified pretraining strategy, improving generalization across modalities and diseases.
• EyeCLIP shows effective transfer to both ocular and systemic disease tasks beyond ophthalmology.