[Paper Review] iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer

Fengtao Zhou, Yingxue Xu+12 more|arXiv (Cornell University)|Apr 1, 2024

Gastric Cancer Management and Outcomes4 citations

TL;DR

This paper proposes iMD4GC, a novel framework for incomplete multimodal data integration in gastric cancer (GC) that combines unimodal attention, cross-modal interaction, and 'more-to-fewer' knowledge distillation to improve treatment response prediction and survival analysis. It achieves 80.2% AUC on GastricRes and 71.4% C-index on GastricSur, significantly outperforming existing methods on large-scale, real-world GC datasets with missing modality patterns.

ABSTRACT

Gastric cancer (GC) is a prevalent malignancy worldwide, ranking as the fifth most common cancer with over 1 million new cases and 700 thousand deaths in 2020. Locally advanced gastric cancer (LAGC) accounts for approximately two-thirds of GC diagnoses, and neoadjuvant chemotherapy (NACT) has emerged as the standard treatment for LAGC. However, the effectiveness of NACT varies significantly among patients, with a considerable subset displaying treatment resistance. Ineffective NACT not only leads to adverse effects but also misses the optimal therapeutic window, resulting in lower survival rate. However, existing multimodal learning methods assume the availability of all modalities for each patient, which does not align with the reality of clinical practice. The limited availability of modalities for each patient would cause information loss, adversely affecting predictive accuracy. In this study, we propose an incomplete multimodal data integration framework for GC (iMD4GC) to address the challenges posed by incomplete multimodal data, enabling precise response prediction and survival analysis. Specifically, iMD4GC incorporates unimodal attention layers for each modality to capture intra-modal information. Subsequently, the cross-modal interaction layers explore potential inter-modal interactions and capture complementary information across modalities, thereby enabling information compensation for missing modalities. To evaluate iMD4GC, we collected three multimodal datasets for GC study: GastricRes (698 cases) for response prediction, GastricSur (801 cases) for survival analysis, and TCGA-STAD (400 cases) for survival analysis. The scale of our datasets is significantly larger than previous studies. The iMD4GC achieved impressive performance with an 80.2% AUC on GastricRes, 71.4% C-index on GastricSur, and 66.1% C-index on TCGA-STAD, significantly surpassing other compared methods.

Motivation & Objective

• To address the challenge of incomplete multimodal data in gastric cancer (GC) prediction, where not all patients have all data types (e.g., imaging, pathology, genomics).
• To improve the accuracy of neoadjuvant chemotherapy (NACT) response prediction and survival prognosis in locally advanced GC (LAGC) patients with heterogeneous treatment responses.
• To develop a robust, interpretable, and scalable deep learning framework that compensates for missing modalities using knowledge distillation and cross-modal interaction.
• To construct and release three large-scale, real-world multimodal GC datasets: GastricRes (698 cases), GastricSur (801 cases), and TCGA-STAD (400 cases) for response and survival prediction.

Proposed method

• iMD4GC uses unimodal attention layers to extract and represent intra-modal features from clinical, radiology, pathology, and genetic data.
• Cross-modal interaction layers are introduced to model inter-modal relationships and enable information compensation when modalities are missing.
• A 'more-to-fewer' knowledge distillation strategy transfers knowledge from models trained on complete data (more modalities) to those with fewer modalities, enhancing robustness to incomplete data.
• The framework employs a 2-layer Transformer architecture with shared 200-dimensional embeddings across modalities after modality-specific feature extraction and projection.
• Hyperparameters α (feature-level distillation) and β (response-level distillation) are tuned per dataset to balance distillation losses and optimize performance.
• The model is trained using Adam optimizer with cosine annealing learning rate scheduling, weight decay, and fixed random seeds for reproducibility.

Experimental results

Research questions

• RQ1 Can a multimodal deep learning framework effectively predict NACT response in gastric cancer patients when some modalities are missing?
• RQ2 How does cross-modal interaction and knowledge distillation improve performance on incomplete multimodal data compared to standard fusion methods?
• RQ3 What is the optimal fusion strategy for multimodal GC data, and does it vary across different prediction tasks (classification vs. regression)?
• RQ4 To what extent does knowledge distillation enhance model generalization and robustness in low-data or incomplete data regimes for GC prognosis?
• RQ5 How do hyperparameters α and β affect performance across different GC datasets and prediction tasks?

Key findings

• iMD4GC achieved 80.2% AUC on the GastricRes dataset for NACT response prediction, significantly outperforming baseline methods.
• On the GastricSur dataset, iMD4GC achieved a C-index of 71.4%, demonstrating strong performance in survival prognosis with incomplete multimodal data.
• On the TCGA-STAD dataset, iMD4GC achieved a C-index of 66.1%, showing consistent performance across independent, large-scale GC cohorts.
• The ablation study confirmed that feature concatenation was the best fusion strategy across all datasets, while low-rank tensor fusion and early fusion with Transformer also showed strong potential.
• Hyperparameter analysis revealed that feature-level distillation (α) was critical for classification (NACT response), while response-level distillation (β) was more important for regression (survival analysis).
• The framework demonstrated inherent interpretability, enabling transparent analysis of modality contributions and decision-making processes for clinical insight.