[Paper Review] Multimodal Deep Learning for Dynamic and Static Neuroimaging: Integrating MRI and fMRI for Alzheimer Disease Analysis

Anima Kujur, Zahra Monfared|arXiv (Cornell University)|Mar 9, 2026

Functional Brain Connectivity Studies0 citations

TL;DR

The paper presents a multimodal deep learning framework combining MRI (3D CNN) and fMRI (Time-Distributed CNN + LSTM/GRU) to classify AD, MCI, and Normal Cognitive State, with data augmentation improving performance on a small paired dataset.

ABSTRACT

Magnetic Resonance Imaging (MRI) provides detailed structural information, while functional MRI (fMRI) captures temporal brain activity. In this work, we present a multimodal deep learning framework that integrates MRI and fMRI for multi-class classification of Alzheimer Disease (AD), Mild Cognitive Impairment, and Normal Cognitive State. Structural features are extracted from MRI using 3D convolutional neural networks, while temporal features are learned from fMRI sequences using recurrent architectures. These representations are fused to enable joint spatial-temporal learning. Experiments were conducted on a small paired MRI-fMRI dataset (29 subjects), both with and without data augmentation. Results show that data augmentation substantially improves classification stability and generalization, particularly for the multimodal 3DCNN-LSTM model. In contrast, augmentation was found to be ineffective for a large-scale single-modality MRI dataset. These findings highlight the importance of dataset size and modality when designing augmentation strategies for neuroimaging-based AD classification.

Motivation & Objective

• Leverage structural (MRI) and functional (fMRI) neuroimaging to improve AD, MCI, and Normal Cognitive State classification.
• Develop a unified spatial-temporal fusion framework for multimodal data.
• Assess the impact of dataset size and data augmentation on model generalization in neuroimaging-based AD analysis.

Proposed method

• Extract spatial features from MRI using a 3D CNN encoder.
• Extract temporal features from fMRI with a Time-Distributed 3D CNN plus LSTM/GRU.
• Fuse MRI and fMRI representations into a shared latent space for classification.
• Train multimodal architectures (3DCNN-LSTM, 3DCNN-GRU) with Adam optimizer (lr=1e-5) for 20 epochs.
• Compare with single-modality baselines and analyze augmentation impact.
• Evaluate using accuracy, AUC, confusion matrices, and other metrics.

Experimental results

Research questions

• RQ1 Can a spatial-temporal fusion of MRI and fMRI improve AD/MCI/NCS classification compared with single modalities?
• RQ2 Does domain-specific data augmentation improve generalization on small paired MRI–fMRI datasets?
• RQ3 Which recurrent architecture (LSTM vs GRU) provides better temporal modeling for fMRI within a multimodal framework?

Key findings

• Augmentation substantially improved multimodal model performance on the small HD dataset (3DCNN-LSTM achieved higher AUCs with augmentation).
• Without augmentation, multimodal models showed limited generalization and notable class confusion, especially for MCI.
• Among recurrent variants, 3DCNN-LSTM generally yielded higher AUC for AD/MCI/NCS than 3DCNN-GRU on augmented data, though GRU offered competitive results with lower compute.
• On a larger MRI-only Kaggle dataset, augmentation sometimes degraded performance, highlighting dataset-size and modality effects on augmentation efficacy.
• The best-performing multimodal model (3DCNN–LSTM) achieved AUCs of 0.76 (AD), 0.86 (MCI), and 0.92 (NCS) on the augmented dataset (HD).
• Findings emphasize the importance of modality fusion and dataset characteristics in designing augmentation strategies for neuroimaging-based AD classification.