[Paper Review] UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities

Muhammad Uzair Khattak, Shahina Kunhimon+3 more|arXiv (Cornell University)|Dec 13, 2024

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

This paper introduces UniMed-CLIP, a unified vision-language model pre-trained on a large-scale, open-source dataset of 5.3 million image-text pairs across six medical imaging modalities. By leveraging Large Language Models to synthesize high-quality captions from label-only datasets, UniMed-CLIP achieves state-of-the-art zero-shot performance, outperforming existing models—including those trained on proprietary data—by up to +12.61 accuracy points while using only one-third of the training data.

ABSTRACT

Vision-Language Models (VLMs) trained via contrastive learning have achieved notable success in natural image tasks. However, their application in the medical domain remains limited due to the scarcity of openly accessible, large-scale medical image-text datasets. Existing medical VLMs either train on closed-source proprietary or relatively small open-source datasets that do not generalize well. Similarly, most models remain specific to a single or limited number of medical imaging domains, again restricting their applicability to other modalities. To address this gap, we introduce UniMed, a large-scale, open-source multi-modal medical dataset comprising over 5.3 million image-text pairs across six diverse imaging modalities: X-ray, CT, MRI, Ultrasound, Pathology, and Fundus. UniMed is developed using a data-collection framework that leverages Large Language Models (LLMs) to transform modality-specific classification datasets into image-text formats while incorporating existing image-text data from the medical domain, facilitating scalable VLM pretraining. Using UniMed, we trained UniMed-CLIP, a unified VLM for six modalities that significantly outperforms existing generalist VLMs and matches modality-specific medical VLMs, achieving notable gains in zero-shot evaluations. For instance, UniMed-CLIP improves over BiomedCLIP (trained on proprietary data) by an absolute gain of +12.61, averaged over 21 datasets, while using 3x less training data. To facilitate future research, we release UniMed dataset, training codes, and models at https://github.com/mbzuai-oryx/UniMed-CLIP.

Motivation & Objective

• To address the scarcity of large-scale, open-access medical image-text datasets for vision-language pretraining.
• To overcome the limitations of modality-specific models by developing a unified, multi-modal vision-language foundation model.
• To enable scalable, data-centric pretraining using LLMs to convert label-only medical datasets into high-quality image-text pairs.
• To release a fully open-source dataset, training code, and models to accelerate public research in medical AI.

Proposed method

• Developed a data-collection framework that uses Large Language Models (LLMs) to convert modality-specific image-label datasets into standardized image-text pairs using medical-style templates.
• Constructed UniMed, a 5.3 million image-text dataset spanning six medical imaging modalities: X-ray, CT, MRI, ultrasound, pathology, and fundus imaging.
• Fine-tuned a vision-language model based on MetaCLIP’s ViT-B/16 vision encoder and BioMed-BERT text encoder, using a contrastive learning objective.
• Trained UniMed-CLIP using a multi-node setup with 16 A100 40GB GPUs, employing a batch size of 128 per GPU and a learning rate of 5e-5 with 2k warmup steps.
• Applied prompt engineering with GPT-4o to generate consistent, clinically appropriate captions from disease labels, ensuring high caption quality.
• Combined synthetic image-text pairs with existing open medical VLM data to enhance diversity and representation quality.

Experimental results

Research questions

• RQ1 Can a unified vision-language model achieve strong zero-shot generalization across diverse medical imaging modalities using only open-source data?
• RQ2 How does the performance of a model trained on synthetic, LLM-generated image-text pairs compare to models trained on proprietary or small-scale datasets?
• RQ3 To what extent can LLMs effectively convert label-only medical datasets into high-quality, clinically relevant image-text pairs for pretraining?
• RQ4 Does a unified model trained on multiple modalities outperform modality-specific models in zero-shot and linear probing evaluations?
• RQ5 Can a data-centric approach using LLM-augmented data achieve state-of-the-art performance with significantly reduced data requirements?

Key findings

• UniMed-CLIP achieved an absolute gain of +12.61 accuracy over BiomedCLIP—trained on closed-source data—when averaged across 21 diverse medical datasets, despite using only one-third of the training data.
• On the PCam histopathology dataset, UniMed-CLIP reached 85.97% accuracy in 100% linear probing, outperforming BiomedCLIP (83.40%) and PMC-CLIP (81.03%).
• In the ACL MRI dataset, UniMed-CLIP achieved 97.28% accuracy in 100% linear probing, surpassing BiomedCLIP (83.84%) and CLIP (81.09%).
• On the ODIR-2x300 fundus imaging dataset, UniMed-CLIP achieved 95.00% accuracy in 100% linear probing, outperforming BiomedCLIP (94.17%) and CLIP (91.73%).
• UniMed-CLIP maintained strong performance on class-imbalanced datasets, including CT, MRI, and retinal fundus imaging, demonstrating robust generalization under skewed data distributions.
• The model achieved state-of-the-art zero-shot performance across all six modalities, with notable improvements on challenging datasets like MediMeTA (CT) and FIVES (fundus), where it outperformed prior models by significant margins.