[Paper Review] Transcriptomics-guided Slide Representation Learning in Computational Pathology

Guillaume Jaume, Lukas Oldenburg+6 more|arXiv (Cornell University)|May 19, 2024

Gene expression and cancer classificationBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes Tangle, a self-supervised multimodal framework that leverages transcriptomics data to guide whole-slide image (WSI) representation learning via contrastive learning between histopathology slides and gene expression profiles. By pre-training on 8,629 human and rat (S+E) pairs across liver, breast, and lung tissues, Tangle achieves state-of-the-art few-shot classification and slide retrieval performance, significantly outperforming supervised and self-supervised baselines on independent test cohorts of 1,265 breast, 1,946 lung, and 4,584 liver WSIs.

ABSTRACT

Self-supervised learning (SSL) has been successful in building patch embeddings of small histology images (e.g., 224x224 pixels), but scaling these models to learn slide embeddings from the entirety of giga-pixel whole-slide images (WSIs) remains challenging. Here, we leverage complementary information from gene expression profiles to guide slide representation learning using multimodal pre-training. Expression profiles constitute highly detailed molecular descriptions of a tissue that we hypothesize offer a strong task-agnostic training signal for learning slide embeddings. Our slide and expression (S+E) pre-training strategy, called Tangle, employs modality-specific encoders, the outputs of which are aligned via contrastive learning. Tangle was pre-trained on samples from three different organs: liver (n=6,597 S+E pairs), breast (n=1,020), and lung (n=1,012) from two different species (Homo sapiens and Rattus norvegicus). Across three independent test datasets consisting of 1,265 breast WSIs, 1,946 lung WSIs, and 4,584 liver WSIs, Tangle shows significantly better few-shot performance compared to supervised and SSL baselines. When assessed using prototype-based classification and slide retrieval, Tangle also shows a substantial performance improvement over all baselines. Code available at https://github.com/mahmoodlab/TANGLE.

Motivation & Objective

• To address the challenge of learning robust, generalizable slide embeddings from giga-pixel whole-slide images (WSIs) without relying on costly pathologist annotations.
• To explore whether gene expression profiles—offering high-resolution molecular context—can serve as a strong, task-agnostic supervisory signal for slide representation learning.
• To develop a multimodal contrastive learning framework that aligns visual slide features with molecular expression embeddings, improving generalization across diverse tissues and species.
• To evaluate the performance of the proposed method on downstream tasks such as few-shot classification, prototype-based classification, and slide retrieval across independent test datasets.
• To analyze the interpretability and quality of learned representations through attention visualization and rank analysis of the embedding space.

Proposed method

• Tangle employs modality-specific encoders: a vision transformer for slide patches and a feed-forward network for gene expression profiles.
• The framework uses symmetric contrastive learning to align slide and expression embeddings by maximizing their mutual similarity in a shared latent space.
• Pre-training is performed on 8,629 (S+E) pairs from The Cancer Genome Atlas (TCGA) and TG-GATEs, covering human and rat tissues across liver, breast, and lung.
• The model is fine-tuned via linear probing on downstream tasks using only a few labeled examples per class, enabling few-shot learning.
• Ablation studies evaluate the contribution of different loss components, including symmetric contrastive loss, L1, and L2 objectives, as well as the use of average vs. random patch views.
• The method incorporates a hybrid contrastive objective combining local-global and local-local alignment to improve feature robustness and generalization.

Experimental results

Research questions

• RQ1 Can transcriptomics data serve as a strong, task-agnostic supervisory signal for self-supervised whole-slide image representation learning?
• RQ2 How does multimodal contrastive learning between histopathology slides and gene expression profiles improve few-shot classification and slide retrieval compared to purely visual SSL or supervised baselines?
• RQ3 To what extent do the learned representations in Tangle exhibit interpretability, such as attention focusing on clinically relevant regions like tumors, without explicit training for these tasks?
• RQ4 What is the impact of different contrastive loss components and hyperparameters (e.g., batch size, temperature, number of patches) on downstream performance?
• RQ5 How does the intrinsic dimensionality of the learned slide embedding space, measured via smooth rank, correlate with downstream classification accuracy?

Key findings

• Tangle achieves significantly better few-shot classification performance than both supervised (ABMIL) and self-supervised (Intra) baselines across three independent test datasets: 1,265 breast WSIs, 1,946 lung WSIs, and 4,584 liver WSIs.
• On the TG-GATEs rat liver lesion classification task, Tangle achieves an AUC of 0.92, outperforming the next best baseline by 7.0 percentage points.
• The symmetric contrastive loss in Tangle leads to a consistent performance boost, while L1 and L2 losses result in a 6.7% and 7.0% drop in AUC, respectively.
• Combining the Tangle objective with an intra-slide contrastive loss (e.g., contrast with average or random view) leads to a performance drop of -2.0% AUC, indicating that the (S+E) signal is sufficient and complementary.
• Attention heatmaps show that Tangle’s frozen ABMIL encoder naturally focuses on tumor regions in both breast and lung WSIs, indicating emergent biological relevance without explicit tumor-related supervision.
• A strong positive correlation is observed between the smooth rank of the slide embedding matrix and downstream classification performance, indicating that Tangle learns a high-dimensional, informative latent space.