[Paper Review] Multimodal contrastive learning for spatial gene expression prediction using histology images

Wenwen Min, Zhiceng Shi+3 more|arXiv (Cornell University)|Jul 11, 2024

Gene expression and cancer classification4 citations

TL;DR

This paper proposes mclSTExp, a multimodal contrastive learning framework using Transformer and DenseNet-121 encoders to predict spatial gene expression from H&E histology images by treating ST spots as 'words' and leveraging self-attention and contrastive learning to integrate spatial and image features. It achieves state-of-the-art performance with 23.01–32.09% higher PCC than prior methods on three datasets, while enabling interpretation of cancer-specific and immune-related genes.

ABSTRACT

In recent years, the advent of spatial transcriptomics (ST) technology has unlocked unprecedented opportunities for delving into the complexities of gene expression patterns within intricate biological systems. Despite its transformative potential, the prohibitive cost of ST technology remains a significant barrier to its widespread adoption in large-scale studies. An alternative, more cost-effective strategy involves employing artificial intelligence to predict gene expression levels using readily accessible whole-slide images (WSIs) stained with Hematoxylin and Eosin (H\&E). However, existing methods have yet to fully capitalize on multimodal information provided by H&E images and ST data with spatial location. In this paper, we propose extbf{mclSTExp}, a multimodal contrastive learning with Transformer and Densenet-121 encoder for Spatial Transcriptomics Expression prediction. We conceptualize each spot as a "word", integrating its intrinsic features with spatial context through the self-attention mechanism of a Transformer encoder. This integration is further enriched by incorporating image features via contrastive learning, thereby enhancing the predictive capability of our model. Our extensive evaluation of extbf{mclSTExp} on two breast cancer datasets and a skin squamous cell carcinoma dataset demonstrates its superior performance in predicting spatial gene expression. Moreover, mclSTExp has shown promise in interpreting cancer-specific overexpressed genes, elucidating immune-related genes, and identifying specialized spatial domains annotated by pathologists. Our source code is available at https://github.com/shizhiceng/mclSTExp.

Motivation & Objective

• To address the high cost of spatial transcriptomics (ST) by enabling accurate prediction of gene expression from readily available, low-cost H&E whole-slide images (WSIs).
• To integrate multimodal information from ST data (spatial gene expression) and H&E images (histological morphology) more effectively than existing methods.
• To improve predictive performance by modeling spatial relationships between ST spots using self-attention and contrastive learning.
• To enable interpretation of biologically relevant genes, including cancer-specific overexpressed genes and immune-related markers.
• To identify spatial domains annotated by pathologists using predicted gene expression patterns.

Proposed method

• Treat each ST spot as a 'word' and model sequences of spots as 'sentences' using a Transformer encoder with learnable positional embeddings.
• Use a DenseNet-121 backbone to extract image features from H&E WSI patches corresponding to each ST spot.
• Apply contrastive learning to align image features and ST spot embeddings in a shared latent space, maximizing similarity for positive pairs (matched image and spot) and minimizing it for negative pairs.
• Integrate spatial context via self-attention in the Transformer, allowing each spot to attend to others and capture long-range spatial dependencies.
• Predict gene expression by aggregating the top-k most similar 'words' (spots) to a query, weighted by their cosine similarity to the image representation.
• Train the model end-to-end using contrastive loss and regression loss on gene expression prediction.

Experimental results

Research questions

• RQ1 Can a multimodal contrastive learning framework effectively combine H&E image features and spatial ST spot data to improve gene expression prediction?
• RQ2 Does modeling ST spots as 'words' in a sequence with self-attention enhance spatial context modeling and prediction accuracy?
• RQ3 Can the model identify biologically meaningful spatial domains and overexpressed genes in cancer tissues?
• RQ4 How does mclSTExp compare to state-of-the-art methods like BLEEP, HisToGene, and STnet in predictive performance across multiple datasets?
• RQ5 To what extent does mclSTExp preserve gene-gene co-expression patterns and biological heterogeneity in predicted data?

Key findings

• mclSTExp achieved a 23.01% higher PCC (ACG) than the second-best method (BLEEP) on the HER2+ breast cancer dataset.
• On the cSCC skin squamous cell carcinoma dataset, mclSTExp showed a 32.09% improvement in PCC (ACG) over BLEEP.
• On the Alex+10x dataset, mclSTExp achieved a 25.56% higher PCC (ACG) than BLEEP, demonstrating consistent superiority across diverse tissue types.
• mclSTExp outperformed all baselines in PCC (HEG) by 27.82% to 36.48% across the three datasets.
• The model successfully predicted HLA-B and HLA-DRA, key immune-related genes previously undetected by other methods, highlighting its biological interpretability.
• Hierarchical clustering of predicted gene expression data preserved known biological co-expression patterns, with correlation matrices closely matching those from actual ST data.