[Paper Review] Multi-modal Graph Fusion for Inductive Disease Classification in Incomplete Datasets

Gerome Vivar, Hendrik Burwinkel+4 more|arXiv (Cornell University)|May 8, 2019

AI in cancer detection3 references4 citations

TL;DR

This paper proposes a multi-modal graph fusion framework using Graph Attention Networks (GATs) to enable inductive, node-level disease classification in incomplete medical datasets with block-wise missing modalities. By training separate GAT branches for MRI, PET, and CSF data and fusing their representations end-to-end, the method achieves superior performance over single-graph baselines and mean-imputation approaches, particularly under high missingness (50%), demonstrating robustness and generalization to out-of-sample patients.

ABSTRACT

Clinical diagnostic decision making and population-based studies often rely on multi-modal data which is noisy and incomplete. Recently, several works proposed geometric deep learning approaches to solve disease classification, by modeling patients as nodes in a graph, along with graph signal processing of multi-modal features. Many of these approaches are limited by assuming modality- and feature-completeness, and by transductive inference, which requires re-training of the entire model for each new test sample. In this work, we propose a novel inductive graph-based approach that can generalize to out-of-sample patients, despite missing features from entire modalities per patient. We propose multi-modal graph fusion which is trained end-to-end towards node-level classification. We demonstrate the fundamental working principle of this method on a simplified MNIST toy dataset. In experiments on medical data, our method outperforms single static graph approach in multi-modal disease classification.

Motivation & Objective

• To address the challenge of multi-modal, incomplete medical data in clinical decision support systems.
• To enable inductive, out-of-sample disease classification despite missing entire modalities (e.g., no MRI or PET scans).
• To improve generalization and robustness over single-graph GNNs that are sensitive to graph initialization.
• To develop an end-to-end trainable fusion mechanism that leverages information across multiple modalities even when some are missing.
• To evaluate the method on real-world medical data (TADPOLE) and a modified MNIST dataset under controlled missingness.

Proposed method

• The method employs multiple GAT branches, each processing one modality (e.g., MRI, PET, CSF) independently, enabling modality-specific representation learning.
• Each modality is processed via a GAT with multi-head attention, allowing dynamic weighting of neighborhood nodes based on feature similarity.
• A late fusion strategy combines the outputs of the individual GAT branches through element-wise averaging or concatenation followed by a final classifier.
• The model is trained end-to-end using a node-level cross-entropy loss, enabling inductive inference on previously unseen patients.
• Graphs are initialized using k-nearest neighbors (NN-Graph) or mutual information (MI-Graph) to capture patient similarity in feature space.
• The approach supports block-wise missingness, where entire modalities can be absent per patient without requiring imputation.

Experimental results

Research questions

• RQ1 Can a multi-modal graph fusion framework achieve better performance than single-graph GNNs in disease classification with incomplete, block-wise missing data?
• RQ2 How does the proposed method perform under varying levels of modality missingness (e.g., 10%, 50%) compared to standard ML and GNN baselines?
• RQ3 Does end-to-end training of multiple GAT branches improve robustness to graph initialization compared to a single static graph?
• RQ4 Can the model generalize to out-of-sample patients not seen during training, even when entire modalities are missing?
• RQ5 Does the fusion of multiple modality-specific graphs outperform mean-imputation strategies in real-world clinical datasets?

Key findings

• On the TADPOLE dataset, the proposed GAT-Fusion method achieved 72.07% accuracy, outperforming the single-graph GAT with mean imputation (GAT-Imp), which scored between 10% and 24% lower.
• With 50% block-wise missingness on the modified MNIST dataset, the GAT-Fusion approach achieved 90.26% accuracy, surpassing Random Forest (92.37%) only at 40% and 50% missingness.
• The GAT-Fusion model demonstrated improved robustness to graph initialization, as it did not rely on a single, potentially suboptimal, graph structure.
• On TADPOLE, the best-performing GAT-Fusion variant (GAT1 with MI-Graph) achieved an AUC of 0.569, outperforming other GAT and baseline models.
• The method successfully enabled inductive inference, classifying patients not included in the training graph, a key advantage over transductive methods.
• The results confirm that multi-graph fusion is more effective than single-graph or mean-imputation baselines in handling real-world incomplete medical data.