[Paper Review] A Transformer-based Diffusion Probabilistic Model for Heart Rate and Blood Pressure Forecasting in Intensive Care Unit

Ping Chang, Huayu Li+5 more|arXiv (Cornell University)|Jan 16, 2023

Hemodynamic Monitoring and TherapyMedicine3 citations

TL;DR

This paper proposes TDSTF, a Transformer-based diffusion probabilistic model that forecasts heart rate and blood pressure in ICU patients using sparse, intermittent vital sign data. By leveraging a triplet-based input mechanism and a Transformer backbone, TDSTF achieves state-of-the-art performance with a 18.9% lower SACRPS and 34.3% lower MSE than the best baseline, while being over 17× faster in inference.

ABSTRACT

Background and Objective: Vital sign monitoring in the Intensive Care Unit (ICU) is crucial for enabling prompt interventions for patients. This underscores the need for an accurate predictive system. Therefore, this study proposes a novel deep learning approach for forecasting Heart Rate (HR), Systolic Blood Pressure (SBP), and Diastolic Blood Pressure (DBP) in the ICU. Methods: We extracted $24,886$ ICU stays from the MIMIC-III database which contains data from over $46$ thousand patients, to train and test the model. The model proposed in this study, Transformer-based Diffusion Probabilistic Model for Sparse Time Series Forecasting (TDSTF), merges Transformer and diffusion models to forecast vital signs. The TDSTF model showed state-of-the-art performance in predicting vital signs in the ICU, outperforming other models' ability to predict distributions of vital signs and being more computationally efficient. The code is available at https://github.com/PingChang818/TDSTF. Results: The results of the study showed that TDSTF achieved a Standardized Average Continuous Ranked Probability Score (SACRPS) of $0.4438$ and a Mean Squared Error (MSE) of $0.4168$, an improvement of $18.9\%$ and $34.3\%$ over the best baseline model, respectively. The inference speed of TDSTF is more than $17$ times faster than the best baseline model. Conclusion: TDSTF is an effective and efficient solution for forecasting vital signs in the ICU, and it shows a significant improvement compared to other models in the field.

Motivation & Objective

• To develop a deep learning model that accurately forecasts heart rate (HR), systolic (SBP), and diastolic (DBP) blood pressure in ICU patients despite sparse and intermittent monitoring.
• To address the limitations of existing models that rely on continuous data and ignore clinical context such as interventions and comorbidities.
• To improve forecasting performance on sparse time series by integrating diffusion probabilistic models with a Transformer-based architecture.
• To enhance computational efficiency for real-time clinical deployment in critical care settings.
• To evaluate the model on a large-scale ICU dataset (MIMIC-III) and demonstrate generalization and robustness across diverse patient populations.

Proposed method

• The model, TDSTF, combines a diffusion probabilistic framework with a Transformer-based backbone to model complex temporal dependencies in sparse vital sign sequences.
• It uses a triplet-based input representation that encodes conditional observations, noise, and time steps, improving signal-to-noise ratio and reducing input disturbance.
• The diffusion process gradually denoises the data through a reverse Markov chain, learning the underlying distribution of vital signs using a denoising U-Net with a Transformer encoder.
• The model is trained end-to-end using a variational lower bound objective, minimizing the reconstruction loss of the denoising process.
• A masking strategy is applied to conditional inputs to prevent information leakage, though the authors note it may not fully exclude invalid data.
• Inference is accelerated by optimizing the number of denoising steps and leveraging the parallel computation capabilities of the Transformer architecture.

Experimental results

Research questions

• RQ1 Can a diffusion probabilistic model effectively capture temporal dependencies in sparse, irregularly sampled ICU vital sign data?
• RQ2 How does the integration of a Transformer backbone improve forecasting performance compared to RNN-based or attention-only models in sparse time series?
• RQ3 To what extent does the triplet-based input design reduce noise and improve signal-to-noise ratio in forecasting sparse vital signs?
• RQ4 How does TDSTF compare in accuracy and inference speed to state-of-the-art models like CSDI, MQ-RNN, and DeepAR on ICU vital sign forecasting tasks?
• RQ5 Can the model detect sudden physiological changes (e.g., tachycardia, hypotension) even when target data is missing, indicating robust representation learning?

Key findings

• TDSTF achieved a Standardized Average Continuous Ranked Probability Score (SACRPS) of 0.4438, representing an 18.9% improvement over the best baseline model.
• The model recorded a Mean Squared Error (MSE) of 0.4168, a 34.3% reduction compared to the best baseline model.
• TDSTF demonstrated over 17 times faster inference speed than the best baseline model, significantly enhancing real-time applicability.
• The model successfully captured both gradual trends and sudden physiological changes in vital signs, even when target conditional data was absent, indicating strong feature interrelation learning.
• The use of a triplet input with 60 time steps improved signal-to-noise ratio and outperformed CSDI, which suffered from residual noise due to less effective masking.
• The model’s performance was robust across HR, SBP, and DBP forecasting, demonstrating generalization across multiple vital signs in the ICU context.