[Paper Review] PLUTO: Pathology-Universal Transformer

Dinkar Juyal, Harshith Padigela+31 more|arXiv (Cornell University)|May 13, 2024

Cancer Genomics and Diagnostics8 citations

TL;DR

PLUTO is a light-weight pathology foundation model pre-trained on 195M image tiles across diverse sites and stains, producing multi-scale embeddings that adapt to slide-, tissue-, and cell-level pathology tasks with task-specific heads. It matches or surpasses task-specific baselines while being more data-efficient and robust to distribution shifts.

ABSTRACT

Pathology is the study of microscopic inspection of tissue, and a pathology diagnosis is often the medical gold standard to diagnose disease. Pathology images provide a unique challenge for computer-vision-based analysis: a single pathology Whole Slide Image (WSI) is gigapixel-sized and often contains hundreds of thousands to millions of objects of interest across multiple resolutions. In this work, we propose PathoLogy Universal TransfOrmer (PLUTO): a light-weight pathology FM that is pre-trained on a diverse dataset of 195 million image tiles collected from multiple sites and extracts meaningful representations across multiple WSI scales that enable a large variety of downstream pathology tasks. In particular, we design task-specific adaptation heads that utilize PLUTO's output embeddings for tasks which span pathology scales ranging from subcellular to slide-scale, including instance segmentation, tile classification, and slide-level prediction. We compare PLUTO's performance to other state-of-the-art methods on a diverse set of external and internal benchmarks covering multiple biologically relevant tasks, tissue types, resolutions, stains, and scanners. We find that PLUTO matches or outperforms existing task-specific baselines and pathology-specific foundation models, some of which use orders-of-magnitude larger datasets and model sizes when compared to PLUTO. Our findings present a path towards a universal embedding to power pathology image analysis, and motivate further exploration around pathology foundation models in terms of data diversity, architectural improvements, sample efficiency, and practical deployability in real-world applications.

Motivation & Objective

• Motivate and address data diversity and robustness challenges in pathology AI due to site-, scanner-, and stain-related variations.
• Develop a universal pathology foundation model (PLUTO) that learns multi-scale WSI representations from diverse data sources.
• Enable efficient adaptation of a single backbone to downstream tasks spanning slide-, tissue-, and cellular/subcellular-level analyses.
• Evaluate PLUTO against state-of-the-art task-specific models and pathology foundation models across multiple benchmarks and modalities.

Proposed method

• Pre-train a light-weight ViT backbone (FlexiViT) on 195M tiles from 50+ sources with four resolutions using self-supervised objectives (DINOv2, iBOT) plus MAE reconstruction.
• Incorporate a Fourier-based reconstruction loss to separately optimize low- and high-frequency content during MAE training.
• Use FlexiViT to allow variable patch sizes and multi-scale masking, enabling efficient multi-resolution pre-training and adaptable inference.
• Extract multi-scale embeddings from tiles and train task-specific adaptation heads (MIL for slide-level, tile classifiers for tissue-level, and instance segmentation heads for cellular/subcellular tasks).
• Adaptation heads include Mask2Former, Mask R-CNN, and other lightweight classifiers, optimized for different data regimes and annotation granularities.

Experimental results

Research questions

• RQ1 Can a single pathology foundation model trained on diverse, multi-site data provide robust embeddings across multiple WSIs scales (slide, tissue, cellular) and staining protocols?
• RQ2 How does PLUTO perform on slide-level classification, tile-level classification, and instance segmentation compared to task-specific baselines and existing pathology FMs?
• RQ3 Does a light-weight backbone with multi-scale masking and Fourier losses improve generalization, sample efficiency, and deployability in real-world pathology workflows?
• RQ4 How do different adaptation heads (MIL, Mask2Former, Mask R-CNN, linear classifiers) influence performance across scales and datasets?

Key findings

• PLUTO matches or exceeds task-specific baselines and other pathology foundation models on diverse benchmarks.
• On NSCLC slide-level subtyping, PLUTO with frozen features achieves 90.2 F1 and 94.0 AUROC in-domain, and 86.1 F1 and 91.2 AUROC out-of-domain, outperforming several baselines.
• On HER2 scoring, PLUTO with frozen features achieves 71.5 F1 and 89.5 AUROC in-domain, and 71.0 F1 and 93.7 AUROC out-of-domain, again surpassing comparable backbones.
• Tile classification on CRC-100K and Camelyon17-WILDS shows strong accuracy with PLUTO embeddings using linear heads (CRC-100K: 96.6% Acc, 95.3% Bal. Acc; Camelyon17-WILDS: 96.2% Acc).
• Gland and nuclei segmentation benchmarks (GlaS, PanNuke) demonstrated competitive performance with PLUTO when paired with appropriate segmentation heads (Mask2Former, Mask R-CNN, etc.).