[Paper Review] CLIP in Medical Imaging: A Survey

Zihao Zhao, Yu-Xiao Liu+9 more|arXiv (Cornell University)|Dec 12, 2023

Radiomics and Machine Learning in Medical Imaging29 citations

TL;DR

This survey analyzes how CLIP is adapted for medical imaging, detailing refined pre-training methods and CLIP-driven applications, challenges, datasets, and future directions.

ABSTRACT

Contrastive Language-Image Pre-training (CLIP), a simple yet effective pre-training paradigm, successfully introduces text supervision to vision models. It has shown promising results across various tasks due to its generalizability and interpretability. The use of CLIP has recently gained increasing interest in the medical imaging domain, serving as a pre-training paradigm for image-text alignment, or a critical component in diverse clinical tasks. With the aim of facilitating a deeper understanding of this promising direction, this survey offers an in-depth exploration of the CLIP within the domain of medical imaging, regarding both refined CLIP pre-training and CLIP-driven applications. In this paper, we (1) first start with a brief introduction to the fundamentals of CLIP methodology; (2) then investigate the adaptation of CLIP pre-training in the medical imaging domain, focusing on how to optimize CLIP given characteristics of medical images and reports; (3) further explore practical utilization of CLIP pre-trained models in various tasks, including classification, dense prediction, and cross-modal tasks; and (4) finally discuss existing limitations of CLIP in the context of medical imaging, and propose forward-looking directions to address the demands of medical imaging domain. Studies featuring technical and practical value are both investigated. We expect this survey will provide researchers with a holistic understanding of the CLIP paradigm and its potential implications. The project page of this survey can also be found on https://github.com/zhaozh10/Awesome-CLIP-in-Medical-Imaging.

Motivation & Objective

• Provide a comprehensive overview of CLIP concepts and variants.
• Analyze how CLIP pre-training is adapted to medical images and reports.
• Summarize CLIP-driven medical imaging applications across tasks.
• Discuss challenges and propose future research directions for medical CLIP.

Proposed method

• Present a taxonomy of CLIP-related medical imaging studies.
• Describe contrastive pre-training objective and zero-shot generalization equations (equations (1)-(4) in CLIP).
• Summarize multi-scale contrast methods like GLoRIA and LoVT and their improvements over global-only CLIP.
• Categorize data-efficient and knowledge-augmented strategies for medical CLIP pre-training.
• Review publicly available medical image-text datasets and associated CLIP models.

Fig. 1: Taxonomy of studies focusing on CLIP in the field of medical imaging.

Experimental results

Research questions

• RQ1 How can CLIP pre-training be adapted to the characteristics of medical images and reports?
• RQ2 What are effective strategies to achieve multi-scale image-text alignment in medical data?
• RQ3 How can data-efficiency and knowledge incorporation improve medical CLIP performance?
• RQ4 What tasks and datasets demonstrate CLIP-driven capabilities in medical imaging?

Key findings

• CLIP's image-text pre-training can be extended to medical imaging, enabling zero-shot domain identification and cross-modal tasks.
• Multi-scale contrast methods (e.g., GLoRIA, LoVT) improve local-text to local-image alignment beyond global-level CLIP, aiding segmentation and detection.
• Data-efficient strategies (correlation-driven contrast, sentence/section-level prompts, knowledge prompts) mitigate small medical datasets and improve robustness.
• A range of datasets (e.g., ROCO, MedICaT, PMC-OA, MIMIC-CXR, PadChest) support medical image-text research and enable pre-trained CLIP models in this domain.
• Variants like GLIP, CLIPSeg, and CRIS extend CLIP to detection and segmentation, informing medical applications such as lesion localization and sentence-grounding.

Fig. 2: The number of medical imaging papers focusing on CLIP has increased rapidly in recent years.