[Paper Review] Time Aggregation and Model Interpretation for Deep Multivariate Longitudinal Patient Outcome Forecasting Systems in Chronic Ambulatory Care

Beau Norgeot, Dmytro Lituiev+2 more|arXiv (Cornell University)|Nov 30, 2018

Machine Learning in Healthcare4 references4 citations

TL;DR

This study proposes a deep learning framework for forecasting rheumatoid arthritis disease activity in chronic ambulatory care using time-aggregated multivariate longitudinal EHR data. It demonstrates that time-distributed dense layers followed by GRUs yield the most generalizable models, and introduces longitudinal permutation importance and confusion plots for clinical interpretability, revealing recent disease activity and steroid use as key predictors.

ABSTRACT

Clinical data for ambulatory care, which accounts for 90% of the nations healthcare spending, is characterized by relatively small sample sizes of longitudinal data, unequal spacing between visits for each patient, with unequal numbers of data points collected across patients. While deep learning has become state-of-the-art for sequence modeling, it is unknown which methods of time aggregation may be best suited for these challenging temporal use cases. Additionally, deep models are often considered uninterpretable by physicians which may prevent the clinical adoption, even of well performing models. We show that time-distributed-dense layers combined with GRUs produce the most generalizable models. Furthermore, we provide a framework for the clinical interpretation of the models.

Motivation & Objective

• To identify optimal time aggregation strategies for deep learning models in longitudinal, ambulatory EHR data with irregular visit spacing and small sample sizes.
• To address clinical resistance to AI by developing interpretable methods for deep time series models in patient outcome forecasting.
• To evaluate the predictive performance of different time aggregation architectures using real-world EHR data from two distinct rheumatology clinics.
• To provide clinical interpretability through longitudinal permutation importance scoring and visualization of model-learned patient representations.

Proposed method

• Employed Bayesian optimization via Hyperas to compare six time aggregation functions: fully dense, time-distributed dense (TDD) followed by dense, GRU, LSTM, causal convolution, and valid-padded convolution.
• Aggregated EHR data into 100-day windows, selecting the most recent value per variable within each window to handle unequal visit intervals and variable data points.
• Used permutation importance scoring over time windows to quantify the contribution of each input variable (e.g., medications, inflammatory markers) to model performance.
• Visualized model-learned patient representations using confusion plots to identify clusters and potential confounding factors across different patient populations.
• Trained models on combined training and validation sets from the University Clinic cohort to produce the final fully trained model for interpretation.
• Applied the Delong method to compute 95% confidence intervals for area under the ROC curve (AUC) to assess statistical significance of performance differences.

Experimental results

Research questions

• RQ1 Which time aggregation function yields the most generalizable and accurate deep learning model for forecasting patient outcomes in longitudinal ambulatory care with irregular visit patterns?
• RQ2 How can deep learning models be made interpretable for clinicians, particularly in the context of multivariate, time-series EHR data?
• RQ3 What are the relative contributions of different clinical variables (e.g., medications, lab values, disease activity scores) over time to model predictions?
• RQ4 How do model-learned patient representations differ between diverse populations, and what do these differences reveal about underlying clinical or social determinants of health?

Key findings

• The time-distributed dense (TDD) layer followed by a GRU achieved the highest AUC of 0.845 (95% CI: [0.753, 0.914]) on the validation set, outperforming other architectures including TDD-LSTM (AUC: 0.838) and TDD-GRU (AUC: 0.845).
• Longitudinal permutation importance scoring revealed that recent time windows (especially the most recent) were most predictive, with CDAI history and steroid prescriptions (e.g., prednisone) being the most influential variables.
• The addition of new steroids at the current visit was a strong predictor of uncontrolled disease, suggesting it may serve as a clinical surrogate for active disease.
• Switching or maintaining a disease-modifying antirheumatic drug (DMARD) strategy at the prior visit was also highly predictive, indicating physician and patient assessment of treatment efficacy.
• Confusion plots showed a single mixed cluster of controlled and uncontrolled patients in the University Clinic cohort, suggesting a natural transition zone, while the Safety Net clinic showed multiple such clusters, indicating potential confounding factors.
• The presence of multiple proximal clusters in the Safety Net cohort may reflect social determinants of health, as this population was predominantly non-White and less likely to have private insurance.