[Paper Review] AliFuse: Aligning and Fusing Multi-modal Medical Data for Computer-Aided Diagnosis

Qiuhui Chen, Yi Hong|arXiv (Cornell University)|Jan 2, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes AliFuse, a transformer-based vision-language model that aligns and fuses multi-modal medical data—specifically MRI scans and non-imaging clinical data like demographics, lab results, and clinical summaries—for improved Alzheimer’s disease diagnosis. By leveraging contrastive learning for cross-modal alignment and cross-attention for feature fusion, AliFuse achieves state-of-the-art performance across five public AD datasets, outperforming eight baselines with a 13%+ accuracy gain over image-only models.

ABSTRACT

Medical data collected for diagnostic decisions are typically multimodal, providing comprehensive information on a subject. While computer-aided diagnosis systems can benefit from multimodal inputs, effectively fusing such data remains a challenging task and a key focus in medical research. In this paper, we propose a transformer-based framework, called Alifuse, for aligning and fusing multimodal medical data. Specifically, we convert medical images and both unstructured and structured clinical records into vision and language tokens, employing intramodal and intermodal attention mechanisms to learn unified representations of all imaging and non-imaging data for classification. Additionally, we integrate restoration modeling with contrastive learning frameworks, jointly learning the high-level semantic alignment between images and texts and the low-level understanding of one modality with the help of another. We apply Alifuse to classify Alzheimer's disease, achieving state-of-the-art performance on five public datasets and outperforming eight baselines.

Motivation & Objective

• To address the challenge of effectively fusing multi-modal medical data—especially imaging and non-imaging clinical data—for improved computer-aided diagnosis.
• To develop a unified framework that aligns vision and language representations into a shared space to enable robust, holistic diagnostic classification.
• To improve diagnostic performance in Alzheimer’s disease classification by integrating diverse data types, including structural MRIs, demographic information, lab results, and clinical narratives.
• To handle numerical values in text more effectively by introducing a specialized loss function to enhance model sensitivity to clinical numbers.

Proposed method

• Convert medical images and non-imaging text data into vision and language tokens for joint processing in a transformer-based architecture.
• Use image-text contrastive learning to align visual and textual embeddings into a shared representation space, enhancing cross-modal understanding.
• Implement cross-attention mechanisms between image and text tokens to enable dynamic, context-aware feature fusion during classification.
• Introduce a dedicated numerical loss ($L_{\text{num}}$) to improve the model’s sensitivity to clinical numbers in text, such as lab values and age.
• Train the model end-to-end on five large public AD datasets (ADNI, NACC, OASIS, AIBL, MIRIAD) using raw MRI volumes and unstructured/structured clinical data.
• Leverage a pre-trained vision encoder and a fine-tuned text encoder to extract and align multi-modal features before classification.

Experimental results

Research questions

• RQ1 Can a unified vision-language model effectively align and fuse multi-modal medical data—including MRI scans and non-imaging clinical data—for improved Alzheimer’s disease diagnosis?
• RQ2 How does explicit cross-modal alignment via contrastive learning impact diagnostic performance compared to simple feature concatenation or late fusion?
• RQ3 What is the relative contribution of different non-imaging data types (e.g., demographics, lab results, clinical summaries) to diagnostic accuracy in Alzheimer’s classification?
• RQ4 To what extent does a specialized loss for numerical values in text improve model performance on clinical data?
• RQ5 How does the model’s attention mechanism reflect clinically relevant features in both images and text?

Key findings

• AliFuse achieves state-of-the-art performance on five public Alzheimer’s disease datasets, with a test accuracy of 87.93% on ADNI, significantly outperforming the image-only baseline by over 13 percentage points.
• The model improves accuracy by more than 6% compared to a text-only baseline and surpasses the SOTA method Irene by over 4% on ADNI.
• The inclusion of non-imaging data, especially lab results, contributes most to performance, with lab results alone improving accuracy by over 3% compared to removing all non-imaging data.
• The ablation study confirms that both alignment loss and the numerical loss ($L_{\text{num}}$) are critical, with the latter alone contributing to over 3% accuracy gain.
• Attention visualization shows that AliFuse focuses on clinically relevant features: brain structures like the ventricle and cortex in images, and key clinical terms such as 'MMSE', 'CDR', 'logical memory', and 'female' in text.
• t-SNE visualization confirms that aligned multi-modal features yield better class separation than unimodal or non-aligned multi-modal representations, with distinct clustering of NC, MCI, and AD groups.