[Paper Review] Leveraging Foundation Models for Content-Based Image Retrieval in Radiology

Stefan Denner, David Zimmerer+9 more|arXiv (Cornell University)|Mar 11, 2024

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper proposes using vision foundation models—particularly weakly-supervised models like BiomedCLIP—as off-the-shelf feature extractors for content-based medical image retrieval (CBIR) in radiology. By benchmarking on a 1.6M-image dataset spanning four modalities and 185 pathologies, it shows that BiomedCLIP achieves a P@1 of 0.594 without fine-tuning, outperforming specialized models and demonstrating strong generalization across diverse pathologies.

ABSTRACT

Content-based image retrieval (CBIR) has the potential to significantly improve diagnostic aid and medical research in radiology. However, current CBIR systems face limitations due to their specialization to certain pathologies, limiting their utility. On the other hand, several vision foundation models have been shown to produce general-purpose visual features. Therefore, in this work, we propose using vision foundation models as powerful and versatile off-the-shelf feature extractors for content-based image retrieval. Our contributions include: (1) benchmarking a diverse set of vision foundation models on an extensive dataset comprising 1.6 million 2D radiological images across four modalities and 161 pathologies; (2) identifying weakly-supervised models, particularly BiomedCLIP, as highly effective, achieving a achieving a P@1 of up to 0.594 (P@3: 0.590, P@5: 0.588, P@10: 0.583), comparable to specialized CBIR systems but without additional training; (3) conducting an in-depth analysis of the impact of index size on retrieval performance; (4) evaluating the quality of embedding spaces generated by different models; and (5) investigating specific challenges associated with retrieving anatomical versus pathological structures. Despite these challenges, our research underscores the vast potential of foundation models for CBIR in radiology, proposing a shift towards versatile, general-purpose medical image retrieval systems that do not require specific tuning. Our code, dataset splits and embeddings are publicly available under https://github.com/MIC-DKFZ/foundation-models-for-cbmir.

Motivation & Objective

• Address the limited generalizability of current CBIR systems, which are typically trained only on a narrow set of pathologies.
• Investigate whether foundation models can serve as versatile, general-purpose feature extractors for medical image retrieval without task-specific fine-tuning.
• Evaluate the performance of diverse foundation models—supervised, self-supervised, and weakly-supervised—on a large-scale, multi-modality, multi-pathology radiology dataset.
• Analyze the challenges in retrieving pathological vs. anatomical structures and assess the impact of index size on retrieval performance.
• Provide a comprehensive benchmark for future research by publicly releasing code, datasets, and embeddings.

Proposed method

• Combined four public datasets (RadImageNet, NIH14, MIMIC, CheXpert) into a unified collection of 1.6 million 2D radiological images across CT, MR, X-ray, and US modalities.
• Utilized 12 vision foundation models—including ResNet, ViT, SAM, MedSAM, CLIP, MedCLIP, BiomedCLIP, MAE, and DINOv2—for feature extraction without fine-tuning.
• Indexed image embeddings using FAISS for efficient similarity search and evaluated retrieval performance using P@1 and mean average precision (mAP).
• Conducted ablation studies by varying the number of indexed samples per class (5 to 3000) to assess performance saturation and clustering behavior.
• Evaluated embedding space quality via kNN classification and linear probing with F1-score and AUPRC to assess clustering and feature relevance.

Experimental results

Research questions

• RQ1 What is the performance of vision foundation models as feature extractors for medical image retrieval?
• RQ2 How does the performance of vision foundation models compare to a specialized, pathology-specific CBIR system?
• RQ3 What effect does the index size have on the retrieval performance of medical CBIR systems?
• RQ4 How well do embedding spaces of vision foundation models capture pathological structures compared to anatomical ones?

Key findings

• BiomedCLIP achieved the highest P@1 of 0.594 on the full dataset, outperforming even specialized models without any fine-tuning.
• Weakly-supervised models like BiomedCLIP and CLIP showed superior generalization, with BiomedCLIP demonstrating the best clustering in the embedding space (kNN F1: 0.667 macro).
• Retrieval performance saturated at around 1000 samples per class in the index, indicating diminishing returns beyond this point.
• Anatomical classes were retrieved more accurately than pathological classes, suggesting that pathological features are harder to distinguish in embedding space.
• CLIP achieved the best linear classification performance (AUPRC: 0.743 micro), indicating strong feature quality for pathology, despite not being medical-specific.
• Self-supervised models like DINOv2 and ViT showed strong linear probe performance, suggesting strong potential for CBIR after fine-tuning with medical data.