[Paper Review] Benchmarking Domain Generalization Algorithms in Computational Pathology

Neda Zamanitajeddin, Mostafa Jahanifar+3 more|arXiv (Cornell University)|Sep 25, 2024

AI in cancer detection4 citations

TL;DR

This study benchmarks 30 domain generalization (DG) algorithms across three computational pathology tasks using a unified, robust framework with extensive cross-validation. Self-supervised learning and stain augmentation consistently outperform other methods, while pretrained foundation models show strong potential, offering practical guidance for selecting DG strategies in histopathology AI.

ABSTRACT

Deep learning models have shown immense promise in computational pathology (CPath) tasks, but their performance often suffers when applied to unseen data due to domain shifts. Addressing this requires domain generalization (DG) algorithms. However, a systematic evaluation of DG algorithms in the CPath context is lacking. This study aims to benchmark the effectiveness of 30 DG algorithms on 3 CPath tasks of varying difficulty through 7,560 cross-validation runs. We evaluate these algorithms using a unified and robust platform, incorporating modality-specific techniques and recent advances like pretrained foundation models. Our extensive cross-validation experiments provide insights into the relative performance of various DG strategies. We observe that self-supervised learning and stain augmentation consistently outperform other methods, highlighting the potential of pretrained models and data augmentation. Furthermore, we introduce a new pan-cancer tumor detection dataset (HISTOPANTUM) as a benchmark for future research. This study offers valuable guidance to researchers in selecting appropriate DG approaches for CPath tasks.

Motivation & Objective

• Address the lack of systematic evaluation of domain generalization (DG) algorithms in computational pathology (CPath).
• Evaluate the effectiveness of 30 DG algorithms across three CPath tasks with varying difficulty using a unified and robust experimental platform.
• Provide actionable recommendations for selecting DG strategies based on empirical performance across diverse domain shift types.
• Introduce a new pan-cancer tumor detection dataset, HISTOPANTUM, to serve as a benchmark for future CPath research.
• Assess the impact of modality-specific techniques (e.g., stain augmentation) and recent advances like pretrained foundation models in DG for pathology.

Proposed method

• Adapt the DomainBed framework to support CPath-specific DG algorithms, including stain normalization and augmentation.
• Use a unified training and evaluation pipeline with 7,560 cross-validation runs across three distinct CPath tasks: binary histology classification, tissue segmentation, and nuclei detection.
• Incorporate modality-specific data augmentation techniques, particularly stain augmentation, to improve robustness to staining variations.
• Evaluate algorithms using the F1 score as a primary metric to ensure robustness to class imbalance.
• Integrate self-supervised learning (SSL) using a ResNet50 model pretrained on TCGA image patches as a strong baseline.
• Include recent SOTA DG methods and pathology-specific approaches (e.g., CausIRL_CORAL, ARM, EQRM) within the benchmarking framework.

Experimental results

Research questions

• RQ1 How do 30 different domain generalization algorithms perform across three diverse computational pathology tasks with varying levels of difficulty?
• RQ2 Which DG strategies—especially self-supervised learning, data augmentation, or model-based invariance learning—yield the most consistent and robust performance in CPath?
• RQ3 To what extent do modality-specific techniques like stain augmentation improve generalization across unseen staining protocols and scanners?
• RQ4 How does the performance of a pretrained foundation model compare to other DG methods in the context of histopathology image analysis?
• RQ5 Which DG algorithms are most robust to hyperparameter sensitivity and show reliable convergence across multiple runs and datasets?

Key findings

• Self-supervised learning (SSL) and stain augmentation consistently outperformed other DG methods across all three CPath tasks, demonstrating strong generalization under domain shift.
• The baseline empirical risk minimization (ERM) approach achieved competitive performance, underscoring the importance of proper baseline implementation and experimental design.
• Algorithms such as ARM, CausIRL_CORAL, Transfer, and EQRM showed high convergence stability and were less sensitive to hyperparameter tuning, making them reliable choices for practical deployment.
• IGA and SANDMask exhibited high sensitivity to hyperparameters and low convergence likelihood, suggesting limited practical utility in CPath applications.
• The proposed HISTOPANTUM dataset provides a new, large-scale benchmark for pan-cancer tumor detection, supporting future research in domain generalization.
• While ResNet50 was used as a fixed feature extractor, the study suggests that replacing it with newer foundation models could further improve performance, indicating a promising direction for future work.