[Paper Review] Multimodal Fusion of Echocardiography and Electronic Health Records for the Detection of Cardiac Amyloidosis

Zishun Feng, Joseph A. Sivak+1 more|arXiv (Cornell University)|Apr 17, 2024

Amyloidosis: Diagnosis, Treatment, OutcomesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This study proposes a transformer-based multimodal fusion model that integrates parasternal long-axis and apical four-chamber echocardiography videos with electronic health record (EHR) data—including demographics, lab tests, and cardiac metrics—to detect cardiac amyloidosis. The method achieves an AUROC of 0.94 on a small dataset of 41 patients, demonstrating that multimodal fusion enables high diagnostic performance with minimal data, outperforming single-modality and larger-data imaging-only approaches.

ABSTRACT

Cardiac amyloidosis, a rare and highly morbid condition, presents significant challenges for detection through echocardiography. Recently, there has been a surge in proposing machine-learning algorithms to identify cardiac amyloidosis, with the majority being imaging-based deep-learning approaches that require extensive data. In this study, we introduce a novel transformer-based multimodal fusion algorithm that leverages information from both imaging and electronic health records. Specifically, our approach utilizes echocardiography videos from both the parasternal long-axis (PLAX) view and the apical 4-chamber (A4C) view along with patients' demographic data, laboratory tests, and cardiac metrics to predict the probability of cardiac amyloidosis. We evaluated our method using 5-fold cross-validation on a dataset comprising 41 patients and achieved an Area Under the Receiver Operating Characteristic curve (AUROC) of 0.94. The experimental results demonstrate that our approach can achieve competitive results with a significantly smaller dataset compared to prior imaging-based methods that required data from thousands of patients. This underscores the potential of leveraging multimodal data to enhance diagnostic accuracy in the identification of complex cardiac conditions such as cardiac amyloidosis.

Motivation & Objective

• To address the challenge of detecting rare and underdiagnosed cardiac amyloidosis in clinical settings where expert interpretation is limited.
• To improve diagnostic accuracy by integrating diverse data sources—echocardiography and structured EHR—beyond imaging alone.
• To evaluate whether multimodal fusion using deep learning can achieve high performance with small datasets, reducing reliance on large-scale imaging data.
• To identify the most informative features across imaging and EHR modalities for cardiac amyloidosis prediction.

Proposed method

• A transformer-based intermediate fusion model combines feature embeddings from two echocardiographic views (PLAX and A4C) and EHR data at a shared latent space.
• Echocardiography features are extracted using a 3D convolutional neural network from video sequences, followed by temporal pooling.
• EHR features are processed via early fusion of demographic, cardiac metric, and lab test data, then embedded via a learned projection layer.
• The model uses multi-head self-attention mechanisms to dynamically weigh contributions from each modality and component.
• Performance is evaluated using 5-fold cross-validation with AUROC as the primary metric.
• An ablation study and attention visualization are used to analyze feature importance and model interpretability.

Experimental results

Research questions

• RQ1 Can multimodal fusion of echocardiography and EHR data improve detection of cardiac amyloidosis compared to single-modality models?
• RQ2 How does intermediate fusion compare to late fusion in combining imaging and EHR data for this task?
• RQ3 Which components of the EHR (demographics, lab tests, cardiac metrics) contribute most to prediction performance?
• RQ4 Can a small dataset of 41 patients yield competitive results compared to larger imaging-only models?
• RQ5 What features are most predictive of cardiac amyloidosis according to the model’s attention mechanism?

Key findings

• The proposed multimodal intermediate fusion model achieved an AUROC of 0.94 on a dataset of 41 patients (17 cases, 24 controls), significantly outperforming single-modality models.
• The model’s performance was superior to a 3D CNN model trained on 2,828 patients (AUROC 0.96) when using only 41 patients, demonstrating data efficiency.
• Lab test results contributed most to EHR performance, with BNP and urine protein levels being among the most important individual features.
• Imaging data contributed more to prediction than EHR data overall, with both PLAX and A4C views providing nearly equal importance.
• The ablation study showed that removing lab test data reduced model performance the most, while demographic and vital signs contributed the least.
• Attention visualization revealed that the model learned to prioritize clinically relevant features, such as wall thickness, ejection fraction, and BNP levels, over less relevant entries.