[Paper Review] Are Natural Domain Foundation Models Useful for Medical Image Classification?

Joana Palés Huix, Ganeshan, Adithya Raju+4 more|arXiv (Cornell University)|Oct 30, 2023

COVID-19 diagnosis using AI4 citations

TL;DR

This study evaluates five state-of-the-art foundation models—DINOv2, CLIP, BLIP, SAM, and SEE-MS—for medical image classification across four benchmark datasets. DINOv2 consistently outperforms ImageNet-1k supervised pretraining, while other models fail to surpass this baseline, indicating that not all foundation models generalize well to medical imaging despite strong performance in natural domains.

ABSTRACT

The deep learning field is converging towards the use of general foundation models that can be easily adapted for diverse tasks. While this paradigm shift has become common practice within the field of natural language processing, progress has been slower in computer vision. In this paper we attempt to address this issue by investigating the transferability of various state-of-the-art foundation models to medical image classification tasks. Specifically, we evaluate the performance of five foundation models, namely SAM, SEEM, DINOv2, BLIP, and OpenCLIP across four well-established medical imaging datasets. We explore different training settings to fully harness the potential of these models. Our study shows mixed results. DINOv2 consistently outperforms the standard practice of ImageNet pretraining. However, other foundation models failed to consistently beat this established baseline indicating limitations in their transferability to medical image classification tasks.

Motivation & Objective

• To assess the transferability of natural domain foundation models to medical image classification tasks.
• To compare the performance of self-supervised and contrastive pretraining models (e.g., DINOv2, CLIP, BLIP) against the standard ImageNet-1k supervised baseline.
• To investigate whether fine-tuning or freezing the backbone improves performance on medical datasets.
• To analyze feature reuse and adaptation across layers in foundation models during medical fine-tuning.
• To determine whether architectural design or pretraining objective influences transfer success in medical imaging.

Proposed method

• Evaluated five foundation models: DINOv2, CLIP (OpenCLIP), BLIP, SAM, and SEE-MS on four medical imaging datasets: ISIC, ChestX-ray14, NIH, and Chexpert.
• Used both linear head and ViT-based classifier heads to assess performance under different adaptation strategies.
• Performed full fine-tuning and partial fine-tuning (freezing or unfreezing specific layers) to analyze feature adaptation.
• Applied k-NN evaluation to assess feature quality and transferability without task-specific adaptation.
• Conducted feature similarity analysis using activation maps to compare representations across models and datasets.
• Trained models using high-resolution inputs and varied dataset sizes to assess data and resolution sensitivity.

Experimental results

Research questions

• RQ1 Can foundation models pre-trained on natural image data outperform standard ImageNet-1k supervised pretraining in medical image classification?
• RQ2 How does fine-tuning versus freezing the backbone affect performance across different foundation models?
• RQ3 To what extent do early vs. late layers in foundation models exhibit transferable features for medical tasks?
• RQ4 Why do some foundation models (e.g., SAM, BLIP) underperform despite strong zero-shot capabilities in natural vision?
• RQ5 How does the pretraining objective (self-supervised vs. contrastive with text) influence transferability to medical imaging?

Key findings

• DINOv2 consistently outperformed ImageNet-1k supervised pretraining across all four medical datasets, demonstrating superior transferability.
• SAM, BLIP, and OpenCLIP failed to surpass the ImageNet-1k baseline in most settings, indicating limited generalization to medical imaging.
• Freezing the foundation model led to performance degradation, especially in later layers, suggesting high-level features are not directly transferable to medical tasks.
• Fine-tuning DINOv2 resulted in faster convergence and higher performance than other models, likely due to fewer layers requiring adaptation.
• Feature similarity analysis showed that early layers reused features across models, while late layers adapted significantly during fine-tuning, indicating task-specific adaptation is essential.
• Appending a DeiT classifier provided only marginal gains, suggesting high-level features from foundation models are not optimal inputs for downstream medical classifiers.