[Paper Review] Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis

Michal Golovanevsky, Carsten Eickhoff+1 more|arXiv (Cornell University)|Jun 17, 2022

Machine Learning in Healthcare32 references130 citations

TL;DR

This paper proposes MADDi, a multimodal deep learning framework that uses cross-modal attention to integrate MRI, genetic (SNP), and structured clinical data for Alzheimer’s disease (AD) and mild cognitive impairment (MCI) diagnosis. By leveraging both self-attention and cross-modal attention mechanisms, MADDi achieves 96.88% test accuracy in three-class classification, demonstrating that clinical data is critical for model performance and that attention mechanisms significantly improve multimodal representation learning over feature concatenation.

ABSTRACT

Alzheimer's Disease (AD) is the most common neurodegenerative disorder with one of the most complex pathogeneses, making effective and clinically actionable decision support difficult. The objective of this study was to develop a novel multimodal deep learning framework to aid medical professionals in AD diagnosis. We present a Multimodal Alzheimer's Disease Diagnosis framework (MADDi) to accurately detect the presence of AD and mild cognitive impairment (MCI) from imaging, genetic, and clinical data. MADDi is novel in that we use cross-modal attention, which captures interactions between modalities - a method not previously explored in this domain. We perform multi-class classification, a challenging task considering the strong similarities between MCI and AD. We compare with previous state-of-the-art models, evaluate the importance of attention, and examine the contribution of each modality to the model's performance. MADDi classifies MCI, AD, and controls with 96.88% accuracy on a held-out test set. When examining the contribution of different attention schemes, we found that the combination of cross-modal attention with self-attention performed the best, and no attention layers in the model performed the worst, with a 7.9% difference in F1-Scores. Our experiments underlined the importance of structured clinical data to help machine learning models contextualize and interpret the remaining modalities. Extensive ablation studies showed that any multimodal mixture of input features without access to structured clinical information suffered marked performance losses. This study demonstrates the merit of combining multiple input modalities via cross-modal attention to deliver highly accurate AD diagnostic decision support.

Motivation & Objective

• To develop a multimodal deep learning framework that improves diagnostic accuracy for Alzheimer’s disease (AD) and mild cognitive impairment (MCI) by integrating diverse data sources.
• To address the limitation of existing multimodal models that simply concatenate features without modeling cross-modal interactions.
• To investigate the contribution of each modality—imaging, genetics, and clinical data—to the overall diagnostic performance.
• To evaluate the impact of attention mechanisms, particularly cross-modal attention, on model performance in multimodal AD classification.
• To demonstrate that structured clinical data plays a pivotal role in contextualizing and improving interpretation of imaging and genetic features.

Proposed method

• MADDi employs a multimodal deep learning architecture that processes MRI, SNP, and structured clinical data through modality-specific neural network backbones.
• The framework applies self-attention to refine features within each modality, enhancing the model's focus on relevant patterns.
• Cross-modal attention is used to explicitly model interactions between different modalities, enabling the network to learn shared representations.
• Features from all modalities are fused via attention-based concatenation, allowing the model to dynamically weigh the importance of each modality during classification.
• The model performs multi-class classification into three groups: controls (CN), mild cognitive impairment (MCI), and Alzheimer’s disease (AD).
• Ablation studies are conducted to evaluate the impact of attention mechanisms and individual modalities on performance, using five random model initializations for robustness.

Experimental results

Research questions

• RQ1 How does cross-modal attention improve multimodal classification performance in Alzheimer’s disease diagnosis compared to simple feature concatenation?
• RQ2 What is the relative contribution of imaging, genetic, and clinical data to the diagnostic accuracy of a multimodal deep learning model?
• RQ3 Does the inclusion of structured clinical data significantly enhance model performance, especially when combined with imaging and genetic data?
• RQ4 How do different attention mechanisms—self-attention and cross-modal attention—affect the model’s ability to distinguish between MCI, AD, and controls?
• RQ5 To what extent does the absence of clinical data degrade performance in multimodal AD diagnosis frameworks?

Key findings

• MADDi achieved a test accuracy of 96.88% across five random model initializations, setting a new state-of-the-art for multimodal three-class AD diagnosis.
• The combination of cross-modal attention and self-attention yielded the highest performance, with a 7.9% improvement in F1-score over models without any attention layers.
• The clinical modality contributed most significantly to model performance, with models lacking clinical data suffering marked performance drops.
• When used in isolation, imaging data produced the best unimodal accuracy (92.28%), but the addition of clinical and genetic data through multimodal learning significantly outperformed unimodal baselines.
• Abllation studies confirmed that models without access to structured clinical data performed substantially worse, highlighting its critical role in contextualizing other modalities.
• The use of cross-modal attention enabled the model to learn richer, more interdependent representations across modalities, improving generalization and diagnostic robustness.