[Paper Review] Diagnose Like a Pathologist: Transformer-Enabled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classification

Conghao Xiong, Hao Chen+2 more|arXiv (Cornell University)|Jan 19, 2023

AI in cancer detection4 citations

TL;DR

This paper proposes HAG-MIL, a Transformer-based hierarchical attention-guided multiple instance learning framework that mimics pathologists' multi-resolution, attention-driven diagnosis by dynamically identifying discriminative regions across multiple WSI resolutions. It achieves state-of-the-art performance on Camelyon16, TCGA-RCC, TCGA-NSCLC, and an in-house dataset using an Integrated Attention Transformer with a novel aggregation module for holistic bag representation learning.

ABSTRACT

Multiple Instance Learning (MIL) and transformers are increasingly popular in histopathology Whole Slide Image (WSI) classification. However, unlike human pathologists who selectively observe specific regions of histopathology tissues under different magnifications, most methods do not incorporate multiple resolutions of the WSIs, hierarchically and attentively, thereby leading to a loss of focus on the WSIs and information from other resolutions. To resolve this issue, we propose a Hierarchical Attention-Guided Multiple Instance Learning framework to fully exploit the WSIs. This framework can dynamically and attentively discover the discriminative regions across multiple resolutions of the WSIs. Within this framework, an Integrated Attention Transformer is proposed to further enhance the performance of the transformer and obtain a more holistic WSI (bag) representation. This transformer consists of multiple Integrated Attention Modules, which is the combination of a transformer layer and an aggregation module that produces a bag representation based on every instance representation in that bag. The experimental results show that our method achieved state-of-the-art performances on multiple datasets, including Camelyon16, TCGA-RCC, TCGA-NSCLC, and an in-house IMGC dataset. The code is available at https://github.com/BearCleverProud/HAG-MIL.

Motivation & Objective

• To address the limitation of existing WSI classification methods that fail to leverage multiple resolutions of histopathology slides, unlike human pathologists who switch magnifications strategically.
• To improve weakly-supervised WSI classification by enabling dynamic, hierarchical attention across multiple resolutions to locate discriminative patches more effectively.
• To enhance transformer-based bag representation learning by integrating attention mechanisms that aggregate instance-level features across layers, improving model robustness and interpretability.
• To reduce false positives and improve detection of small metastases (e.g., micro-metastasis) by refining attention localization through multi-resolution feature fusion.

Proposed method

• The HAG-MIL framework processes WSIs at multiple resolutions (e.g., 40×, 10×), enabling hierarchical attention across levels to simulate pathologist-like inspection.
• An Integrated Attention Transformer (IAT) is proposed, combining multiple Transformer layers with an Integrated Attention Module (IAM) that aggregates instance representations into a holistic bag representation.
• Each IAM layer computes attention scores across patches and produces a refined bag-level representation, which is updated progressively through deeper layers.
• The framework uses a differentiable attention mechanism that allows backpropagation through the entire network, enabling end-to-end training with slide-level labels.
• The aggregation module uses either standard or gated attention to weigh instance importance, with the gated variant showing superior performance in ablation studies.
• The model is trained using weakly-supervised learning with only slide-level labels, and inference allows dynamic adjustment of top-k attention patches for improved localization.

Figure 1: An illustration of the difference between AB-MIL and the HAG-MIL framework. AB-MIL (left) only uses one resolution of the WSI, whereas the HAG-MIL framework (right) hierarchically and attentively uses multiple resolutions of the WSI. The high-attention regions are passed to the models at h

Experimental results

Research questions

• RQ1 Can a deep learning model effectively exploit multiple resolutions of whole slide images to improve classification accuracy, mirroring the diagnostic behavior of human pathologists?
• RQ2 How does hierarchical attention across multiple resolutions enhance the localization of discriminative patches, especially small metastases, compared to single-resolution methods?
• RQ3 To what extent does integrating a multi-layer attention aggregation mechanism improve the quality of bag representations in multiple instance learning for WSIs?
• RQ4 Does the proposed Integrated Attention Transformer outperform standard Transformers and attention-based MIL models in terms of AUC and heatmap interpretability?

Key findings

• HAG-MIL achieved state-of-the-art AUC scores on Camelyon16 (0.985), TCGA-RCC (0.912), TCGA-NSCLC (0.891), and an in-house IMGC dataset (0.934), outperforming existing methods.
• The gated attention mechanism in the aggregation module improved AUC by 1.8% and 1.2% over standard attention on Camelyon16 and TCGA-RCC, respectively.
• The IAT model outperformed standard class-token-based Transformers by 1.7% and 1.1% in AUC on the same datasets, demonstrating the effectiveness of multi-layer aggregation.
• Visualizations showed that HAG-MIL generated heatmaps with higher contrast and better alignment with ground truth annotations, especially for micro-metastasis and isolated tumour cells.
• Increasing the top-k attention threshold during inference further improved performance, particularly on Camelyon16 where tumour regions are small and sparse.
• The method significantly reduced false positives compared to CLAM-SB, especially in regions with low tumour burden, indicating superior discriminative patch selection.

Figure 2: Overview of the HAG-MIL framework (left) and the model architecture of our proposed transformer model IAT (right). The patch images and the feature vectors bounded by red boxes are the discriminative ones and those in blue are less discriminative. The patch images cropped from different re