[Paper Review] MedFMC: A Real-world Dataset and Benchmark For Foundation Model Adaptation in Medical Image Classification

Dequan Wang, Xiaosong Wang+13 more|arXiv (Cornell University)|Jun 16, 2023

COVID-19 diagnosis using AIMedicine3 citations

TL;DR

This paper introduces MedFMC, a real-world medical image classification benchmark with 22,349 images across five diverse clinical tasks—chest X-rays, pathology, endoscopy, neonatal jaundice, and diabetic retinopathy. It evaluates few-shot adaptation of foundation models using prompt-based tuning and fine-tuning, demonstrating that few-shot methods, especially VPT, achieve superior and more stable performance, particularly on rare disease classes, with low variance across 10-shot experiments.

ABSTRACT

Foundation models, often pre-trained with large-scale data, have achieved paramount success in jump-starting various vision and language applications. Recent advances further enable adapting foundation models in downstream tasks efficiently using only a few training samples, e.g., in-context learning. Yet, the application of such learning paradigms in medical image analysis remains scarce due to the shortage of publicly accessible data and benchmarks. In this paper, we aim at approaches adapting the foundation models for medical image classification and present a novel dataset and benchmark for the evaluation, i.e., examining the overall performance of accommodating the large-scale foundation models downstream on a set of diverse real-world clinical tasks. We collect five sets of medical imaging data from multiple institutes targeting a variety of real-world clinical tasks (22,349 images in total), i.e., thoracic diseases screening in X-rays, pathological lesion tissue screening, lesion detection in endoscopy images, neonatal jaundice evaluation, and diabetic retinopathy grading. Results of multiple baseline methods are demonstrated using the proposed dataset from both accuracy and cost-effective perspectives.

Motivation & Objective

• Address the lack of publicly available, diverse, and real-world datasets for evaluating foundation model adaptation in medical image classification.
• Enable benchmarking of few-shot learning methods across multiple medical imaging modalities and clinical tasks to assess generalizability.
• Evaluate cost-effective adaptation techniques—especially prompt-based learning—on limited data, mimicking real clinical scenarios where labeled data is scarce.
• Demonstrate that few-shot methods outperform fine-tuning when data is extremely limited, particularly for rare disease classes.
• Provide a standardized, publicly available dataset and codebase to accelerate research in foundation model adaptation for medical imaging.

Proposed method

• Collected 22,349 real-world 2D medical images from five clinical institutions across diverse modalities: X-ray, pathology, endoscopy, digital photography, and retinography.
• Constructed five distinct classification tasks: multi-label thoracic abnormalities (ChestDR), binary gastrointestinal lesion detection (ColonPath), multi-label colorectal lesion detection (Endo), binary neonatal jaundice evaluation (NeoJaundice), and multi-class diabetic retinopathy grading (Retino).
• Evaluated multiple few-shot learning baselines including Meta-baseline, VPT (Visual Prompt Tuning), and standard fine-tuning using ImageNet-pretrained backbones (e.g., Swin-T, ResNeXt).
• Applied prompt-based tuning (VPT) with learnable visual tokens to adapt foundation models using only 10–20% of the data, minimizing parameter updates.
• Used standard evaluation metrics: accuracy, AUC, mAP, and mean average precision, with 10 repeated experiments to assess variance in few-shot settings.
• Compared few-shot methods against full fine-tuning using 20% of the few-shot pool, and evaluated the impact of data augmentation on performance.

(a) ChestDR

Experimental results

Research questions

• RQ1 Can few-shot adaptation methods generalize effectively across diverse, real-world medical image classification tasks with limited data?
• RQ2 How do prompt-based methods like VPT compare to standard fine-tuning in terms of accuracy, stability, and data efficiency on medical imaging benchmarks?
• RQ3 To what extent do few-shot methods improve classification performance for rare disease classes compared to common classes?
• RQ4 What is the variance in performance across repeated few-shot experiments, and how stable are the results across different data splits?
• RQ5 Does data augmentation significantly improve fine-tuning performance when only a few samples are available?

Key findings

• VPT achieved the best overall performance among few-shot baselines, demonstrating superior generalization across all five medical image classification tasks.
• Few-shot methods, particularly VPT, outperformed fine-tuning when only 10 samples were available, with minimal performance drop compared to full fine-tuning on 20% data.
• The variance in accuracy and AUC across 10 repeated 10-shot experiments was consistently below 5%, indicating high stability of few-shot learning results.
• Rare disease classes (tail classes) in thoracic X-ray classification achieved higher or comparable AUC values using few-shot methods, suggesting improved robustness for low-frequency conditions.
• Fine-tuning with 1–5 samples was often unstable due to overfitting or underfitting, highlighting a critical limitation of conventional fine-tuning in low-data regimes.
• Data augmentation had only a marginal impact on fine-tuning performance, suggesting that data efficiency is more dependent on model adaptation strategy than augmentation in extreme low-shot settings.

(b) ColonPath