[Paper Review] A Clinical Benchmark of Public Self-Supervised Pathology Foundation Models

Gabriele Campanella, Shengjia Chen+11 more|arXiv (Cornell University)|Jul 9, 2024

Ethics in Clinical Research6 citations

TL;DR

The paper benchmarks public pathology foundation models trained with self-supervised learning on a clinical, multi-institution dataset across disease detection and biomarker prediction tasks, comparing performance and resource usage. It analyzes scaling behavior and dataset effects, offering guidance for model selection and future benchmarking.

ABSTRACT

The use of self-supervised learning (SSL) to train pathology foundation models has increased substantially in the past few years. Notably, several models trained on large quantities of clinical data have been made publicly available in recent months. This will significantly enhance scientific research in computational pathology and help bridge the gap between research and clinical deployment. With the increase in availability of public foundation models of different sizes, trained using different algorithms on different datasets, it becomes important to establish a benchmark to compare the performance of such models on a variety of clinically relevant tasks spanning multiple organs and diseases. In this work, we present a collection of pathology datasets comprising clinical slides associated with clinically relevant endpoints including cancer diagnoses and a variety of biomarkers generated during standard hospital operation from two medical centers. We leverage these datasets to systematically assess the performance of public pathology foundation models and provide insights into best practices for training new foundation models and selecting appropriate pretrained models.

Motivation & Objective

• Assess the performance of public pathology foundation models on clinically relevant tasks spanning multiple organs and diseases.
• Compare models trained with different SSL algorithms and datasets to identify best practices for training and model selection.
• Evaluate how model size and training resources relate to downstream clinical task performance.
• Provide a live benchmark and guidance for future model development in computational pathology.

Proposed method

• Embed slide-level tissue tiles (20x) into feature representations using a given foundation model.
• Aggregate tile features with Gated MIL Attention (GMA) and a linear classifier for slide-level predictions.
• Use Monte Carlo Cross-Validation with 20 folds to estimate generalization (80% train, 20% val per fold).
• Train with AdamW optimizer, cosine decay with warm-up, 50 epochs on 1 GPU per task/model.

Figure 1: Benchmarking Results: Detection Tasks.

Experimental results

Research questions

• RQ1 How do public pathology foundation models compare on disease detection and biomarker prediction across organs and cancer types?
• RQ2 What is the impact of pretraining data size and architecture on downstream clinical task performance in pathology?
• RQ3 Do larger models consistently improve performance on biomarker prediction tasks, and how does pretraining dataset composition influence this?
• RQ4 How do computational resources and training cost relate to downstream performance in pathology SSL models?
• RQ5 Can a standardized live benchmark effectively guide model selection for clinical deployment?

Key findings

• DINO and DINOv2 trained models show comparable performance across detection tasks, with ImageNet or CTransPath pretrained encoders underperforming on average.
• For detection tasks, model size shows little to no scaling benefit; smaller models like SP21M/SP22M perform similarly to larger ones on these tasks.
• Biomarker prediction tasks exhibit more variability, with UNI and Prov-GigaPath often outperforming others, especially when tissue representation aligns with pretraining data (e.g., lung).
• Ici outcome (ICI response) prediction remains challenging across models, with AUCs near chance for NSCLC cohorts.
• Pretraining dataset composition can significantly influence biomarker performance, suggesting tissue prevalence in pretraining data affects task outcomes.
• Larger models can help some biomarker tasks, but benefits are task-dependent and potentially confounded by data composition.
• Across all models tested, increasing computational resources did not universally improve downstream task performance for detection or biomarker prediction.

Figure 2: Benchmarking Results: Biomarker Prediction Tasks.