[Paper Review] Towards Precision Healthcare: Robust Fusion of Time Series and Image Data

Ali Rasekh, Reza Heidari+5 more|arXiv (Cornell University)|May 24, 2024

Artificial Intelligence in HealthcareHealth Professions3 citations

TL;DR

This paper proposes a robust multimodal deep learning framework that fuses time series and medical image data using separate encoders, attention-based modality fusion, and an uncertainty loss function for improved mortality prediction and phenotyping. The method achieves superior performance and noise robustness on the MIMIC-IV and MIMIC-CXR datasets, outperforming state-of-the-art models like MedFuse, especially under high noise levels (up to 60%) with only a 2% performance drop.

ABSTRACT

With the increasing availability of diverse data types, particularly images and time series data from medical experiments, there is a growing demand for techniques designed to combine various modalities of data effectively. Our motivation comes from the important areas of predicting mortality and phenotyping where using different modalities of data could significantly improve our ability to predict. To tackle this challenge, we introduce a new method that uses two separate encoders, one for each type of data, allowing the model to understand complex patterns in both visual and time-based information. Apart from the technical challenges, our goal is to make the predictive model more robust in noisy conditions and perform better than current methods. We also deal with imbalanced datasets and use an uncertainty loss function, yielding improved results while simultaneously providing a principled means of modeling uncertainty. Additionally, we include attention mechanisms to fuse different modalities, allowing the model to focus on what's important for each task. We tested our approach using the comprehensive multimodal MIMIC dataset, combining MIMIC-IV and MIMIC-CXR datasets. Our experiments show that our method is effective in improving multimodal deep learning for clinical applications. The code will be made available online.

Motivation & Objective

• To address the challenge of integrating heterogeneous clinical data—specifically time series and medical images—for improved predictive modeling in healthcare.
• To enhance model robustness in noisy, real-world clinical environments where data quality varies.
• To improve performance in multi-label classification tasks such as phenotyping by modeling uncertainty in predictions.
• To enable dynamic, task-aware fusion of modalities through attention mechanisms for better interpretability and adaptability.
• To develop a flexible, scalable framework applicable to diverse clinical decision support tasks beyond mortality prediction.

Proposed method

• Utilizes separate deep neural network encoders for time series data (e.g., vital signs, lab results) and medical images (e.g., chest X-rays) to learn modality-specific representations.
• Employs an attention mechanism to dynamically weigh the importance of each modality during fusion, enabling task-adaptive focus on relevant information.
• Applies an uncertainty loss function in a multi-task learning setup to prioritize easier, more certain classification tasks and improve generalization.
• Introduces a noise augmentation strategy using Gaussian noise on 10–60% of pixels (images) and time steps (time series), simulating real-world data imperfections.
• Preprocesses the MIMIC-IV and MIMIC-CXR datasets by aligning timestamps and standardizing image inputs to ensure coherent multimodal training.
• Trains and evaluates the model under both in-distribution and out-of-distribution noise settings to test robustness in realistic clinical scenarios.

Experimental results

Research questions

• RQ1 How can time series and image data be effectively fused in multimodal deep learning for clinical prediction tasks?
• RQ2 To what extent does attention-based modality fusion improve model performance and robustness under noisy data conditions?
• RQ3 Can an uncertainty-aware loss function enhance performance in multi-label clinical classification tasks with imbalanced data?
• RQ4 How does the proposed model compare to state-of-the-art methods like MedFuse in terms of accuracy and robustness under increasing noise levels?
• RQ5 What is the impact of modality-specific encoding and dynamic attention on interpretability and predictive accuracy in phenotyping and mortality prediction?

Key findings

• The proposed model achieves superior robustness under noisy conditions, with only a 2% performance drop at 60% noise levels, outperforming MedFuse, which shows significant degradation.
• The attention-based fusion mechanism enables dynamic, task-relevant attention allocation, improving model adaptability and predictive accuracy across diverse clinical tasks.
• The uncertainty loss function improves performance in multi-label classification by prioritizing simpler, more certain tasks, leading to better overall generalization.
• The model demonstrates strong performance on both mortality prediction and phenotyping tasks, including chronic kidney disease, liver disease, and post-procedural complications.
• The integration of modality-specific encoders and attention-based fusion leads to enhanced discriminative power, as evidenced by improved AUC and F1 scores on benchmark MIMIC datasets.
• The framework maintains high reliability even when tested on noisy data without prior exposure to noise during training, reflecting strong real-world applicability.