[Paper Review] GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series

Edward De Brouwer, Jaak Simm+2 more|arXiv (Cornell University)|May 29, 2019

Machine Learning in Healthcare44 references119 citations

TL;DR

GRU-ODE-Bayes integrates a continuous-time GRU (GRU-ODE) with a Bayesian observation updater (GRU-Bayes) to model sporadically observed multivariate time series, outperforming state-of-the-art methods on healthcare and climate data and encoding a continuity prior.

ABSTRACT

Modeling real-world multidimensional time series can be particularly challenging when these are sporadically observed (i.e., sampling is irregular both in time and across dimensions)-such as in the case of clinical patient data. To address these challenges, we propose (1) a continuous-time version of the Gated Recurrent Unit, building upon the recent Neural Ordinary Differential Equations (Chen et al., 2018), and (2) a Bayesian update network that processes the sporadic observations. We bring these two ideas together in our GRU-ODE-Bayes method. We then demonstrate that the proposed method encodes a continuity prior for the latent process and that it can exactly represent the Fokker-Planck dynamics of complex processes driven by a multidimensional stochastic differential equation. Additionally, empirical evaluation shows that our method outperforms the state of the art on both synthetic data and real-world data with applications in healthcare and climate forecast. What is more, the continuity prior is shown to be well suited for low number of samples settings.

Motivation & Objective

• Motivate and address irregular, sporadic sampling in multidimensional time series.
• Develop a continuous-time GRU variant (GRU-ODE) to evolve latent dynamics between observations.
• Introduce a Bayesian update network (GRU-Bayes) to assimilate sporadic observations.
• Show that the model encodes a continuity prior and can represent Fokker-Planck dynamics for SDEs.
• Evaluate performance against baselines on real-world healthcare and climate datasets.

Proposed method

• Derive a GRU-based ODE (GRU-ODE) that propagates latent state h(t) between observations using a Lipschitz continuous dynamics.
• Introduce GRU-Bayes to update h(t) at observation times via a Bayesian-like update, handling partial observation masks.
• Combine GRU-ODE and GRU-Bayes into GRU-ODE-Bayes, producing an ODE with discrete jumps at observation times.
• Define a loss with pre-jump negative log-likelihood and post-jump KL-divergence to encourage Bayesian-like updates.
• Handle Gaussian and binomial/multinomial observations with analytical loss forms where possible.
• Support various ODE solvers (Euler, explicit midpoint, Dormand-Prince) and discuss computational trade-offs.

Experimental results

Research questions

• RQ1 Can GRU-ODE-Bayes accurately model the continuous latent dynamics of sporadically observed multidimensional time series?
• RQ2 Does integrating sporadic inputs into the ODE dynamics improve forecasting and capture inter-variable dynamics?
• RQ3 Can the approach recover or approximate the Fokker-Planck dynamics of underlying SDEs (e.g., Ornstein-Uhlenbeck) exactly in some cases?
• RQ4 How does GRU-ODE-Bayes compare against state-of-the-art baselines on real-world healthcare and climate datasets, especially in low-data regimes?

Key findings

• GRU-ODE-Bayes outperforms multiple baselines (NeuralODE-VAE, GRU-D, T-LSTM, etc.) on both healthcare and climate forecasting tasks in terms of MSE and NegLL.
• The continuity prior encoded by GRU-ODE provides a performance edge, particularly in small-sample regimes.
• The model can exactly represent Fokker-Planck dynamics for multivariate Ornstein-Uhlenbeck processes and can handle nonlinear SDEs.
• Jointly modeling continuous latent dynamics and sporadic observations enables learning of inter-variable correlations and responsive updates when new data arrives.
• In experiments, GRU-ODE-Bayes achieves the best or near-best metrics across datasets, indicating robust handling of irregular sampling.