[Paper Review] Multimodal Prototyping for cancer survival prediction

Andrew H. Song, Richard J. Chen+4 more|arXiv (Cornell University)|Jun 28, 2024

Biomedical Text Mining and Ontologies6 citations

TL;DR

MMP introduces unsupervised morphological and pathway prototypes to compress histology and transcriptomics tokens, enabling memory-efficient multimodal fusion with Transformer or OT cross-alignment for cancer survival prediction across six TCGA cohorts.

ABSTRACT

Multimodal survival methods combining gigapixel histology whole-slide images (WSIs) and transcriptomic profiles are particularly promising for patient prognostication and stratification. Current approaches involve tokenizing the WSIs into smaller patches (>10,000 patches) and transcriptomics into gene groups, which are then integrated using a Transformer for predicting outcomes. However, this process generates many tokens, which leads to high memory requirements for computing attention and complicates post-hoc interpretability analyses. Instead, we hypothesize that we can: (1) effectively summarize the morphological content of a WSI by condensing its constituting tokens using morphological prototypes, achieving more than 300x compression; and (2) accurately characterize cellular functions by encoding the transcriptomic profile with biological pathway prototypes, all in an unsupervised fashion. The resulting multimodal tokens are then processed by a fusion network, either with a Transformer or an optimal transport cross-alignment, which now operates with a small and fixed number of tokens without approximations. Extensive evaluation on six cancer types shows that our framework outperforms state-of-the-art methods with much less computation while unlocking new interpretability analyses.

Motivation & Objective

• Motivate prognostication by integrating histology (WSIs) and transcriptomics for survival prediction.
• Reduce token dimensionality via prototype-based summaries to enable efficient multimodal fusion.
• Develop a unified fusion framework (Transformer or OT) operating on compact prototype tokens.
• Enable interpretability by visualizing interactions between morphological and pathway prototypes.
• Demonstrate predictive performance gains across multiple cancer types on TCGA data.

Proposed method

• Construct morphological prototypes by clustering patch embeddings with Gaussian mixtures to produce a small, fixed slide summary (C_h <= 32).
• Represent transcriptomics via fixed biological pathway prototypes (Hallmark pathways, C_g = 50).
• Match token dimensions across modalities using learned projections before fusion.
• Fuse modalities with either Transformer attention or entropic-regularized Optimal Transport cross-alignment, enabling intra- and cross-modal interactions.
• Predict survival using Cox proportional hazards loss on the fused patient embedding.
• Optionally incorporate prototype-specific encodings and prototype-specific post-attention networks to enhance prototype interpretability.

Figure 1: Overview of MMP . (A) The tessellated WSI patches (tokens) are projected to low-dimensional embeddings with a pretrained patch encoder. The patch embeddings ( $N_{\text{h.}}>10^{4}$ ) are aggregated to slide summary using a small set of prototypes ( $C_{\text{h.}}<$ 32). (B) The transcript

Experimental results

Research questions

• RQ1 Can morphology-based prototypes effectively summarize WSIs for survival prediction without loss of prognostic information?
• RQ2 Can pathway-based prototypes condense transcriptomic data into a fixed set of tokens suitable for multimodal fusion?
• RQ3 How do Transformer-based and OT-based cross-modal fusion compare when using prototype tokens?
• RQ4 Does prototype-based multimodal fusion outperform existing uni- and multimodal baselines across multiple cancer types?
• RQ5 To what extent do prototype interactions enable interpretable cross-modal prognostic insights?

Key findings

• MMP outperforms nearly all baselines (both unimodal and multimodal) across six TCGA cancer types.
• Prototype-based histology summarization achieves large input compression (>300x) while preserving predictive power.
• Pathway prototypes provide compact, meaningful representations of transcriptomics and enable effective multimodal fusion.
• Transformer and OT cross-alignment yield comparable performance, validating the connection between the two approaches.
• MMP variants with prototypes achieve higher average concordance index than clinical baselines and many baselines (e.g., MMP_OT and MMP_Trans. show strong results).
• Prototype-based unimodal baselines (AttnMISL, unimodal MMP) remain among the best performers, underscoring prototype quality.

Figure 2: Cross-modal interaction visualization . (A) A WSI for a BRCA patient. (B) The morphological prototype heatmap for $c=13$ (C13), representing invasive ductal carcinoma (IDC), based on the posterior distribution for C13. (C) Prototype assignment map showing the closest morphological prototyp