[Paper Review] Phikon-v2, A large and public feature extractor for biomarker prediction

Alexandre Filiot, Paul Jacob+2 more|arXiv (Cornell University)|Sep 13, 2024

Machine Learning in Bioinformatics17 citations

TL;DR

Phikon-v2 is a large ViT-L self-supervised feature extractor trained on 460M histology tiles from 30+ cancer sites, achieving competitive biomarker-prediction performance and outperforming several baselines in multi-cohort evaluation. Ensembling further boosts downstream AUC significantly.

ABSTRACT

Gathering histopathology slides from over 100 publicly available cohorts, we compile a diverse dataset of 460 million pathology tiles covering more than 30 cancer sites. Using this dataset, we train a large self-supervised vision transformer using DINOv2 and publicly release one iteration of this model for further experimentation, coined Phikon-v2. While trained on publicly available histology slides, Phikon-v2 surpasses our previously released model (Phikon) and performs on par with other histopathology foundation models (FM) trained on proprietary data. Our benchmarks include eight slide-level tasks with results reported on external validation cohorts avoiding any data contamination between pre-training and evaluation datasets. Our downstream training procedure follows a simple yet robust ensembling strategy yielding a +1.75 AUC increase across tasks and models compared to one-shot retraining (p<0.001). We compare Phikon (ViT-B) and Phikon-v2 (ViT-L) against 14 different histology feature extractors, making our evaluation the most comprehensive to date. Our result support evidences that DINOv2 handles joint model and data scaling better than iBOT. Also, we show that recent scaling efforts are overall beneficial to downstream performance in the context of biomarker prediction with GigaPath and H-Optimus-0 (two ViT-g with 1.1B parameters each) standing out. However, the statistical margins between the latest top-performing FMs remain mostly non-significant; some even underperform on specific indications or tasks such as MSI prediction - deposed by a 13x smaller model developed internally. While latest foundation models may exhibit limitations for clinical deployment, they nonetheless offer excellent grounds for the development of more specialized and cost-efficient histology encoders fueling AI-guided diagnostic tools.

Motivation & Objective

• Motivate the need for large-scale, in-domain self-supervised histology pre-training to improve biomarker prediction.
• Scale data and model size to assess scaling benefits for histology foundation models.
• Provide a public, non-commercial feature extractor for researchers to benchmark and build upon.
• Evaluate Phikon-v2 across multiple external cohorts to test generalization and data leakage controls.

Proposed method

• Pre-train a ViT-L model on 460,060,584 tiles extracted from 58,359 publicly available WSIs spanning 30+ cancer sites using DINOv2.
• Use a tissue-mask based tile extraction pipeline at 20x magnification (224x224 tiles) with at least 60% tissue content.
• Fine-tune and evaluate via a two-step MIL setup: extract patch features with a frozen extractor, then aggregate with ABMIL for slide-level predictions.
• Employ 5-case label-stratified splits and 5-fold cross-validation per task, training ensembles of 25 ABMIL models, followed by external-validation inference ensembles.
• Compare Phikon-v2 (ViT-L) to Phikon (ViT-B) and 14 other histology encoders across 8 downstream tasks, using bootstrapped AUC and Fisher-permutation tests.

Experimental results

Research questions

• RQ1 Does scaling both pre-training data and model size with DINOv2 yield superior downstream performance compared to smaller baselines and previous Phikon models?
• RQ2 How does Phikon-v2 perform relative to other public histology encoders across diverse, externally held biomarker tasks?
• RQ3 What is the impact of ensembling versus one-shot retraining on slide-level biomarker predictions?
• RQ4 Are there consistent performance trends across tasks and datasets that inform deployment of histology foundation models?

Key findings

• Phikon-v2 (ViT-L) shows competitive performance with state-of-the-art histology foundation models across 8 external slide-level tasks.
• Ensembling predictions yields a +1.75 to +1.75 AUC increase on average over one-shot retraining across tasks (p<0.001).
• Compared to Phikon and 14 other extractors, ViT-L+ models (including Phikon-v2) generally rank in top tiers, with some task-dependent variation.
• DINOv2-based scaling generally benefits downstream biomarker prediction, though margins between top models are often non-significant across tasks.
• Specialized, smaller models (e.g., iBOT-based MSI predictor) can outperform larger FMs on specific tasks, underscoring that scaling alone does not guarantee best biomarker performance.
• Ensembling provides robust gains and should be favored in benchmarks and practical deployments for weakly supervised histology tasks.