[Paper Review] Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction

Guillaume Jaume, Anurag Vaidya+4 more|arXiv (Cornell University)|Apr 13, 2023

Cancer-related molecular mechanisms research9 citations

TL;DR

SurvPath learns pathway-based tokens from transcriptomics and histology patch tokens from WSIs, fusing them with a memory-efficient Transformer to predict disease-specific survival on TCGA data and provide multimodal interpretability.

ABSTRACT

Integrating whole-slide images (WSIs) and bulk transcriptomics for predicting patient survival can improve our understanding of patient prognosis. However, this multimodal task is particularly challenging due to the different nature of these data: WSIs represent a very high-dimensional spatial description of a tumor, while bulk transcriptomics represent a global description of gene expression levels within that tumor. In this context, our work aims to address two key challenges: (1) how can we tokenize transcriptomics in a semantically meaningful and interpretable way?, and (2) how can we capture dense multimodal interactions between these two modalities? Specifically, we propose to learn biological pathway tokens from transcriptomics that can encode specific cellular functions. Together with histology patch tokens that encode the different morphological patterns in the WSI, we argue that they form appropriate reasoning units for downstream interpretability analyses. We propose fusing both modalities using a memory-efficient multimodal Transformer that can model interactions between pathway and histology patch tokens. Our proposed model, SURVPATH, achieves state-of-the-art performance when evaluated against both unimodal and multimodal baselines on five datasets from The Cancer Genome Atlas. Our interpretability framework identifies key multimodal prognostic factors, and, as such, can provide valuable insights into the interaction between genotype and phenotype, enabling a deeper understanding of the underlying biological mechanisms at play. We make our code public at: https://github.com/ajv012/SurvPath.

Motivation & Objective

• Motivate integrating bulk transcriptomics with whole-slide images (WSIs) to improve patient survival prognostication.
• Propose a semantically meaningful transcriptomics tokenizer based on biological pathways for end-to-end learning.
• Develop a memory-efficient Transformer fusion to model dense pathway–patch interactions.
• Provide a multi-level interpretability framework to uncover unimodal and cross-modal prognostic factors.
• Demonstrate state-of-the-art performance on five TCGA datasets compared to unimodal and other multimodal baselines.

Proposed method

• Tokenize transcriptomics into biological pathway tokens using a sparse MLP mapping from gene expression to pathway representations.
• Extract histology patch tokens from WSIs using a pre-trained Swin Transformer and project to a common token dimension.
• Fuse pathway and patch tokens with a memory-efficient Transformer that decouples pathway–pathway, pathway–patch, and patch–pathway interactions while ignoring patch–patch interactions.
• Predict survival using a discrete-time hazard model with a negative log-likelihood survival loss under censorship.
• Provide an interpretability framework using Integrated Gradients for pathways/genes and Transformer attention for cross-modal interactions (pathway↔patch).

Experimental results

Research questions

• RQ1 Can pathway-based tokenization of transcriptomics capture meaningful cellular processes for multimodal prognostication?
• RQ2 Can a memory-efficient Transformer model model dense interactions between pathway tokens and histology patch tokens to improve survival prediction?
• RQ3 Does early fusion of pathway and patch tokens outperform late fusion baselines in survival analysis?
• RQ4 What unimodal and cross-modal prognostic insights can be derived from the model’s attention and attribution analyses?

Key findings

• SurvPath achieves state-of-the-art concordance (c-index) on five TCGA datasets when compared to unimodal and multimodal baselines.
• Pathway-based transcriptomics tokens (331 pathways) provide semantically meaningful and learnable units that align with histology features for prognosis.
• The memory-efficient attention mechanism enables modeling of pathway–patch and pathway–pathway interactions at scale without prohibitive memory costs.
• Survival predictions benefit from early fusion of modalities, outperforming several baseline fusion strategies.
• Interpretability analyses identify key multimodal prognostic factors and illustrate how specific pathways interact with morphological patterns.