[Paper Review] Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review

Asim Waqas, Aakash Tripathi+3 more|arXiv (Cornell University)|Mar 11, 2023

Computational Drug Discovery Methods4 citations

TL;DR

This review synthesizes state-of-the-art deep learning approaches—particularly Graph Neural Networks (GNNs) and Transformers—for multimodal data integration in oncology, demonstrating their potential to enhance cancer diagnosis, prognosis, and treatment personalization by fusing diverse data types such as genomics, radiology, pathology, and clinical records. The key contribution is a comprehensive analysis of current methods, challenges, and future directions for scalable, interpretable, and robust multimodal oncology models.

ABSTRACT

Cancer has relational information residing at varying scales, modalities, and resolutions of the acquired data, such as radiology, pathology, genomics, proteomics, and clinical records. Integrating diverse data types can improve the accuracy and reliability of cancer diagnosis and treatment. There can be disease-related information that is too subtle for humans or existing technological tools to discern visually. Traditional methods typically focus on partial or unimodal information about biological systems at individual scales and fail to encapsulate the complete spectrum of the heterogeneous nature of data. Deep neural networks have facilitated the development of sophisticated multimodal data fusion approaches that can extract and integrate relevant information from multiple sources. Recent deep learning frameworks such as Graph Neural Networks (GNNs) and Transformers have shown remarkable success in multimodal learning. This review article provides an in-depth analysis of the state-of-the-art in GNNs and Transformers for multimodal data fusion in oncology settings, highlighting notable research studies and their findings. We also discuss the foundations of multimodal learning, inherent challenges, and opportunities for integrative learning in oncology. By examining the current state and potential future developments of multimodal data integration in oncology, we aim to demonstrate the promising role that multimodal neural networks can play in cancer prevention, early detection, and treatment through informed oncology practices in personalized settings.

Motivation & Objective

• To analyze the state-of-the-art in multimodal learning (MML) using deep neural networks, specifically GNNs and Transformers, in oncology.
• To identify key challenges in integrating heterogeneous oncology data across genomics, imaging, pathology, and clinical records.
• To highlight limitations such as class imbalance, modality collapse, oversmoothing in GNNs, and lack of explainability in multimodal models.
• To propose a roadmap for developing scalable, interpretable, and generalizable multimodal deep learning frameworks for precision oncology.
• To compile and provide access to a centralized, periodically updated list of publicly available oncology datasets for research use.

Proposed method

• Systematic review of recent literature on multimodal data fusion using deep neural networks in oncology.
• Focus on Graph Neural Networks (GNNs) and Transformers as core architectures for integrating heterogeneous data modalities.
• Analysis of architectural components such as attention mechanisms in Transformers and message-passing in GNNs for multimodal feature learning.
• Evaluation of techniques for handling data challenges, including data augmentation, regularization, and federated learning for privacy.
• Application of explainability tools like GNNExplainer and SubgraphX to interpret model decisions in multimodal settings.
• Use of foundation models such as CLIP, FLAVA, and GPT-4 as benchmarks for multimodal representation learning in oncology.

Figure 1: Number of publications involving deep learning, graph neural networks (GNNs), GNNs in the medical domain, overall multimodal and multimodal in biomedical and clinical sciences in the period 2014-2023 [ 3 ] .

Experimental results

Research questions

• RQ1 How do GNNs and Transformers enable effective fusion of multimodal oncology data, including genomics, imaging, and clinical records?
• RQ2 What are the primary technical and clinical challenges in deploying multimodal deep learning models in oncology, such as class imbalance and modality collapse?
• RQ3 How can explainability and trustworthiness be improved in multimodal GNNs and Transformers for clinical decision support?
• RQ4 What role do federated learning and privacy-preserving techniques play in training multimodal models across distributed oncology data sources?
• RQ5 What are the key open problems in scalability, uncertainty quantification, and generalization for multimodal oncology models?

Key findings

• GNNs and Transformers have shown strong performance in oncology tasks such as tumor classification, prognosis prediction, and treatment response assessment by integrating multimodal data.
• The RadGenNets model successfully fused PET scans, genomics, and clinical data using CNNs and dense networks to predict gene mutations in NSCLC.
• Modality collapse remains a critical issue, where models over-rely on dominant modalities and underutilize others, despite architectural efforts to mitigate it.
• Oversmoothing in GNNs limits performance in deep architectures, especially when training over many layers, though techniques like skip-connections and dropout help reduce the effect.
• Explainability tools such as GNNExplainer and SubgraphX are emerging but remain underdeveloped for multimodal GNNs and Transformers in oncology.
• Federated learning enables training on distributed oncology data while preserving privacy, offering a viable path for large-scale model development across institutions.

Figure 2: From population to single cell, cancer data modalities at scales.