[Paper Review] Latent ODEs for Irregularly-Sampled Time Series

Yulia Rubanova, Ricky T. Q. Chen+1 more|arXiv (Cornell University)|Jul 8, 2019

Time Series Analysis and Forecasting18 references151 citations

TL;DR

The paper introduces ODE-RNNs and Latent ODEs to model irregularly sampled time series with continuous-time latent dynamics, improving predictions over standard RNNs, and extends Latent ODEs with a Poisson process for observation times.

ABSTRACT

Time series with non-uniform intervals occur in many applications, and are difficult to model using standard recurrent neural networks (RNNs). We generalize RNNs to have continuous-time hidden dynamics defined by ordinary differential equations (ODEs), a model we call ODE-RNNs. Furthermore, we use ODE-RNNs to replace the recognition network of the recently-proposed Latent ODE model. Both ODE-RNNs and Latent ODEs can naturally handle arbitrary time gaps between observations, and can explicitly model the probability of observation times using Poisson processes. We show experimentally that these ODE-based models outperform their RNN-based counterparts on irregularly-sampled data.

Motivation & Objective

• Motivate and model irregularly-sampled time series with continuous-time hidden dynamics.
• Generalize RNN transitions to neural ODEs to handle varying observation intervals.
• Integrate Latent ODEs with an ODE-RNN recognition network for better inference on sparse data.
• Incorporate Poisson process likelihood for observation times to capture informative measurement patterns.
• Evaluate against RNN-based baselines and on real-world irregularly-sampled datasets.

Proposed method

• Define hidden dynamics between observations with an ODE: h'(t)=f_theta(h(t),t).
• Update at observations with an RNNCell: h_i = RNNCell(h_i', x_i).
• Train autoregressive ODE-RNNs and Latent ODEs under a variational autoencoder framework.
• Use ODE solvers with adjoint methods for memory-efficient gradients.
• Optionally model observation times via a Poisson process with rate lambda(t) depending on latent state z(t).
• Compare autoregressive (ODE-RNN) and latent-variable (Latent ODE) architectures including encoder variants (RNN encoder vs. ODE-RNN encoder).

Experimental results

Research questions

• RQ1 Can continuous-time latent dynamics (ODE-RNN and Latent ODE) improve modeling of irregularly-sampled time series over traditional RNNs?
• RQ2 Does using an ODE-based recognition network improve latent-variable inference for sparse data?
• RQ3 Does augmenting Latent ODEs with a Poisson-process likelihood improve modeling of observation times?
• RQ4 How do these ODE-based models perform on interpolation and extrapolation tasks across synthetic and real-world datasets?
• RQ5 What are the computational trade-offs relative to standard RNNs and RNN-based VAEs?

Key findings

• ODE-RNNs outperform standard RNNs on irregularly-sampled data, especially with sparser observations.
• Latent ODEs with an ODE-RNN encoder yield better extrapolation and interpolation than RNN-enc-VAEs.
• Latent ODEs provide explicit latent uncertainty and interpretable dynamics via z_0 and f_theta(z).
• Incorporating a Poisson process likelihood for observation times is feasible but did not always improve downstream tasks like mortality classification in PhysioNet.
• On MuJoCo data, Latent ODEs with an ODE encoder achieve the best interpolation/extrapolation MSE; autoregressive models struggle with extrapolation.
• Latent ODEs exhibit latent space that correlates with physical parameters and show decreasing posterior entropy as more observations are seen.