[Paper Review] Directly Modeling Missing Data in Sequences with RNNs: Improved Classification of Clinical Time Series

Zachary C. Lipton, David C. Kale+1 more|arXiv (Cornell University)|Jun 13, 2016

Machine Learning in Healthcare39 citations

TL;DR

This paper proposes directly modeling missing data in clinical time series using RNNs with binary indicators for missingness, significantly improving AUC and F1 scores in multilabel diagnosis classification. It demonstrates that RNNs outperform linear models and MLPs by learning complex dependencies from missingness patterns, which can be more predictive than test results themselves for certain diseases.

ABSTRACT

We demonstrate a simple strategy to cope with missing data in sequential inputs, addressing the task of multilabel classification of diagnoses given clinical time series. Collected from the intensive care unit (ICU) of a major urban medical center, our data consists of multivariate time series of observations. The data is irregularly sampled, leading to missingness patterns in re-sampled sequences. In this work, we show the remarkable ability of RNNs to make effective use of binary indicators to directly model missing data, improving AUC and F1 significantly. However, while RNNs can learn arbitrary functions of the missing data and observations, linear models can only learn substitution values. For linear models and MLPs, we show an alternative strategy to capture this signal. Additionally, we evaluate LSTMs, MLPs, and linear models trained on missingness patterns only, showing that for several diseases, what tests are run can be more predictive than the results themselves.

Motivation & Objective

• To address the challenge of irregularly sampled, multivariate clinical time series with missing data in ICU settings.
• To evaluate whether RNNs can effectively model missingness patterns as signals rather than noise.
• To compare the performance of RNNs, LSTMs, MLPs, and linear models when trained on missingness patterns alone.
• To investigate whether the pattern of which tests are performed is more predictive than the actual test values for certain diagnoses.
• To develop a simple yet effective strategy for handling missing data in sequential clinical data without imputation.

Proposed method

• Using binary indicators to represent missingness in re-sampled clinical time series, treating missing values as informative features.
• Training RNNs and LSTMs on sequences augmented with binary masks indicating which observations are missing.
• Evaluating models on multilabel diagnosis classification using AUC and F1 as metrics.
• Training separate models using only missingness patterns (i.e., which tests were performed) to assess predictive power independent of values.
• Comparing RNNs’ ability to learn arbitrary functions of missing data and observations against linear models’ reliance on substitution values.
• Applying standard re-sampling to irregularly spaced ICU data to create fixed-length sequences for modeling.

Experimental results

Research questions

• RQ1 Can RNNs effectively learn from missingness patterns in clinical time series without imputation?
• RQ2 How does modeling missing data directly with binary indicators compare to standard imputation or substitution in classification performance?
• RQ3 To what extent can the pattern of which tests are performed predict diagnoses, independent of test results?
• RQ4 Do RNNs outperform linear models and MLPs when learning from missing data signals?
• RQ5 Is the structure of missingness itself a meaningful signal for multilabel clinical diagnosis classification?

Key findings

• RNNs trained with binary indicators for missingness significantly improve AUC and F1 scores in multilabel clinical diagnosis classification.
• The pattern of which tests are performed can be more predictive of certain diagnoses than the actual test results, especially when modeled directly.
• Linear models and MLPs benefit from the same binary missingness indicators, but are limited to learning substitution-based signals, unlike RNNs.
• LSTMs, MLPs, and linear models trained solely on missingness patterns achieve meaningful performance, indicating that test ordering and availability carry diagnostic information.
• The proposed method avoids imputation and directly models missingness as a signal, leading to improved model robustness and performance on irregularly sampled clinical data.
• RNNs demonstrate a superior ability to learn complex, non-linear relationships between missingness and clinical outcomes compared to linear baselines.