[Paper Review] Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks

Mohammed Baharoon, Waseem Qureshi+4 more|arXiv (Cornell University)|Dec 4, 2023

Radiomics and Machine Learning in Medical Imaging8 citations

TL;DR

This study evaluates DINOv2, a self-supervised general-purpose vision foundation model, on radiology tasks (classification and segmentation) across X-ray, CT, and MRI, comparing to supervised, self-supervised, and weakly-supervised baselines and exploring few-shot and parameter-efficient fine-tuning.

ABSTRACT

The integration of deep learning systems into healthcare has been hindered by the resource-intensive process of data annotation and the inability of these systems to generalize to different data distributions. Foundation models, which are models pre-trained on large datasets, have emerged as a solution to reduce reliance on annotated data and enhance model generalizability and robustness. DINOv2 is an open-source foundation model pre-trained with self-supervised learning on 142 million curated natural images that exhibits promising capabilities across various vision tasks. Nevertheless, a critical question remains unanswered regarding DINOv2's adaptability to radiological imaging, and whether its features are sufficiently general to benefit radiology image analysis. Therefore, this study comprehensively evaluates the performance DINOv2 for radiology, conducting over 200 evaluations across diverse modalities (X-ray, CT, and MRI). To measure the effectiveness and generalizability of DINOv2's feature representations, we analyze the model across medical image analysis tasks including disease classification and organ segmentation on both 2D and 3D images, and under different settings like kNN, few-shot learning, linear-probing, end-to-end fine-tuning, and parameter-efficient fine-tuning. Comparative analyses with established supervised, self-supervised, and weakly-supervised models reveal DINOv2's superior performance and cross-task generalizability. The findings contribute insights to potential avenues for optimizing pre-training strategies for medical imaging and enhancing the broader understanding of DINOv2's role in bridging the gap between natural and radiological image analysis. Our code is available at https://github.com/MohammedSB/DINOv2ForRadiology

Motivation & Objective

• Motivate the use of foundation models to reduce annotated data needs in medical imaging and improve cross-domain generalization.
• Systematically evaluate DINOv2 on diverse radiology benchmarks covering 2D and 3D data, spanning classification and segmentation tasks.
• Assess performance under multiple evaluation settings (kNN, linear-probing, few-shot, end-to-end fine-tuning, PEFT).
• Compare DINOv2 to supervised, self-supervised, and weakly-supervised baselines to gauge relative strengths across tasks.

Proposed method

• Use DINOv2 ViT-g/14 and ViT-L/14 (and smaller distilled variants) as frozen backbones for feature extraction.
• Evaluate on 9 public radiology benchmarks spanning X-ray, CT, and MRI for classification and segmentation.
• Assess out-of-the-box performance with lightweight heads; additionally test end-to-end fine-tuning and parameter-efficient fine-tuning (LoRA, BitFit).
• Compare against supervised, self-supervised, and weakly-supervised models across standardized evaluation protocols (AUROC for classification, Dice/Jaccard for segmentation).
• For segmentation, compare frozen encoder with linear vs U-Net decoders; for classification, analyze kNN, linear probing, few-shot, and fine-tuning.
• Standardize training and tuning pipelines to isolate model performance and enable fair cross-model comparisons.

Figure 2: PCA component visualization. Following [ 8 ] , the PCA is computed between patches of images that are in the same column, and the first 3 components are shown. This is done for X-ray, CT, and MRI scans. Thresholding is used on the first component to remove the background.

Experimental results

Research questions

• RQ1 Can DINOv2 representations generalize across both disease classification and organ segmentation in radiology?
• RQ2 How does DINOv2 compare to supervised, self-supervised, and weakly-supervised baselines on radiology benchmarks?
• RQ3 What is the impact of few-shot, linear-probing, and end-to-end vs parameter-efficient fine-tuning for DINOv2 in medical imaging tasks?
• RQ4 Do PEFT methods (LoRA, BitFit) enable competitive performance with far fewer trainable parameters when adapting DINOv2 to radiology tasks?
• RQ5 How does DINOv2 perform across 2D (X-ray) and 3D (CT, MRI) modalities and different organ/ disease benchmarks?

Key findings

• DINOv2 generally outperforms self-supervised and weakly-supervised baselines on classification tasks and is competitive with supervised ImageNet21K-pretrained models.
• On segmentation, DINOv2 outperforms weakly-supervised methods by a large margin and is competitive with or surpasses end-to-end supervised methods in many tasks.
• DINOv2 demonstrates cross-task generalizability, often outperforming supervised models on segmentation and SAM on classification in certain settings.
• Parameter-efficient fine-tuning (LoRA, BitFit) with DINOv2 ViT-g/14 achieves competitive results using less than 1% of total parameters, approaching end-to-end fine-tuning performance.
• Compared to SAM, DINOv2 yields superior classification performance on radiology datasets; cross-task generalizability is highlighted across tasks and modalities.

Figure 3: Linear probing for disease classification compared to self-supervised and weakly-supervised methods. The figure shows the performance of linear probing DINOv2 compared to other self-supervised and weakly-supervised models.