[Paper Review] Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancers

Charlie Saillard, Olivier Dehaene+5 more|arXiv (Cornell University)|Sep 13, 2021

Genetic factors in colorectal cancerMedicine19 citations

TL;DR

This study introduces a self-supervised learning (SSL) approach using MoCo V2 to pretrain deep neural networks on histology images from The Cancer Genome Atlas (TCGA), significantly improving detection of dMMR/MSI status in colorectal and gastric cancers. The method achieves state-of-the-art AUCs of 0.92 and 0.83 for CRC and gastric tumors, respectively, with strong generalization to external cohorts and improved identification of biologically relevant histological patterns.

ABSTRACT

Microsatellite instability (MSI) is a tumor phenotype whose diagnosis largely impacts patient care in colorectal cancers (CRC), and is associated with response to immunotherapy in all solid tumors. Deep learning models detecting MSI tumors directly from H&E stained slides have shown promise in improving diagnosis of MSI patients. Prior deep learning models for MSI detection have relied on neural networks pretrained on ImageNet dataset, which does not contain any medical image. In this study, we leverage recent advances in self-supervised learning by training neural networks on histology images from the TCGA dataset using MoCo V2. We show that these networks consistently outperform their counterparts pretrained using ImageNet and obtain state-of-the-art results for MSI detection with AUCs of 0.92 and 0.83 for CRC and gastric tumors, respectively. These models generalize well on an external CRC cohort (0.97 AUC on PAIP) and improve transfer from one organ to another. Finally we show that predictive image regions exhibit meaningful histological patterns, and that the use of MoCo features highlighted more relevant patterns according to an expert pathologist.

Motivation & Objective

• To improve the accuracy of dMMR/MSI detection from H&E-stained whole slide images using deep learning.
• To overcome the limitations of ImageNet-pretrained models in medical imaging by leveraging self-supervised learning on histology data.
• To enhance model generalization across different cancer types and external datasets.
• To validate that self-supervised features highlight biologically meaningful histological patterns relevant to MSI.
• To enable broader, cost-effective MSI screening in cancers with low MSI prevalence where routine testing is not standard.

Proposed method

• Pretrained a ResNet-50 backbone using MoCo V2 on H&E-stained whole slide images from TCGA, without any labels.
• Trained the SSL model on 1,000+ CRC and 500+ gastric cancer slides from TCGA, using contrastive learning to learn generalizable features.
• Fine-tuned linear classifiers on top of frozen MoCo-learned features for MSI/MSS classification using MeanPool, Chowder, and DeepMIL architectures.
• Compared performance against ImageNet-pretrained models across multiple datasets and evaluation protocols, including cross-validation and external cohort testing.
• Used attention-based MIL models (DeepMIL, Chowder) to aggregate tile-level features into whole-slide predictions.
• Validated feature interpretability by analyzing attention maps and comparing them to expert-annotated regions of interest.

Experimental results

Research questions

• RQ1 Can self-supervised learning on histology images outperform ImageNet pretraining for MSI detection in colorectal and gastric cancers?
• RQ2 Does MoCo-based feature learning improve generalization to unseen cohorts, such as the PAIP CRC dataset?
• RQ3 Can SSL features highlight biologically relevant histological patterns associated with MSI, as confirmed by expert pathologists?
• RQ4 How does the performance of SSL-based models compare to state-of-the-art models relying on ImageNet pretraining in cross-cancer transfer learning?
• RQ5 To what extent does pretraining on histology data alone improve transferability across different cancer types?

Key findings

• The MoCo V2-pretrained model achieved an AUC of 0.92 on TCGA colorectal cancer (TCGA-CRC), outperforming ImageNet-pretrained models by +0.07 in cross-validation.
• On gastric cancer (TCGA-Gastric), the MoCo model reached an AUC of 0.83, a +0.06 improvement over ImageNet baselines.
• The model generalized robustly to an external CRC cohort (PAIP), achieving an AUC of 0.97, indicating strong transferability.
• When jointly pretraining on CRC and gastric cancer data (MoCo-CRC-Gastric), the model achieved AUCs of 0.88 across all three classifiers (MeanPool, Chowder, DeepMIL), demonstrating cross-organ transfer capability.
• Expert pathologists confirmed that attention maps from MoCo features highlighted clinically relevant regions such as tumor-infiltrating lymphocytes and immune-rich stromal areas.
• The MoCo-based features consistently highlighted more relevant and biologically interpretable patterns than features from ImageNet-pretrained models.