[Paper Review] Neural Controlled Differential Equations for Irregular Time Series

Patrick Kidger, James Morrill+2 more|arXiv (Cornell University)|May 18, 2020

Model Reduction and Neural Networks64 references154 citations

TL;DR

Introduces Neural CDEs, a continuous-time model for irregular and partially observed time series that updates via controlled differential equations and supports memory-efficient adjoint backpropagation, achieving state-of-the-art results among ODE/RNN baselines.

ABSTRACT

Neural ordinary differential equations are an attractive option for modelling temporal dynamics. However, a fundamental issue is that the solution to an ordinary differential equation is determined by its initial condition, and there is no mechanism for adjusting the trajectory based on subsequent observations. Here, we demonstrate how this may be resolved through the well-understood mathematics of \emph{controlled differential equations}. The resulting \emph{neural controlled differential equation} model is directly applicable to the general setting of partially-observed irregularly-sampled multivariate time series, and (unlike previous work on this problem) it may utilise memory-efficient adjoint-based backpropagation even across observations. We demonstrate that our model achieves state-of-the-art performance against similar (ODE or RNN based) models in empirical studies on a range of datasets. Finally we provide theoretical results demonstrating universal approximation, and that our model subsumes alternative ODE models.

Motivation & Objective

• Motivate and address limitations of Neural ODEs when data arrive irregularly or partially observed.
• Propose a Neural CDE framework that uses controlled differential equations driven by a data path X.
• Demonstrate memory-efficient adjoint backpropagation across observations.
• Establish universal approximation properties and compare empirically against ODE/RNN baselines.
• Provide practical guidance on implementation and evaluation for irregular time series tasks.

Proposed method

• Define X as a natural cubic spline path through observed data to drive a CDE.
• Formulate z_t = z_{t0} + ∫_{t0}^{t} f_theta(z_s) dX_s with z_{t0} = zeta_theta(x0, t0).
• Use neural networks f_theta and zeta_theta to learn the vector field and initial state.
• Train via adjoint backpropagation with O(H) memory across observations, yielding overall O(L+H) memory.
• Show that a Neural CDE can be written as z_t = z_{t0} + ∫ g_theta,X(z_s, s) ds when X is differentiable, enabling standard ODE solvers.
• Argue that Neural CDEs subsume GRU-ODE-like models and provide universal approximation results.

Experimental results

Research questions

• RQ1 Can a continuous-time model driven by irregular data outperform discrete-time RNN/ODE-based approaches on irregular or partially observed time series?
• RQ2 Does the Neural CDE framework support memory-efficient backpropagation across observations while maintaining competitive accuracy?
• RQ3 Do Neural CDEs provide universal approximation capabilities and subsume alternative ODE-based approaches?
• RQ4 How does the observational intensity channel and partial observation handling affect performance on real-world datasets?

Key findings

• Neural CDEs achieve state-of-the-art accuracy among compared models on irregular/partially observed datasets (CharacterTrajectories, PhysioNet sepsis prediction) and demonstrate superior memory efficiency.
• On CharacterTrajectories with 30%, 50%, and 70% data dropped, Neural CDE attains 98.7% ± 0.8% accuracy, with memory usage about 1.3 MB, outperforming GRU-ODE, GRU-D, ODE-RNN, and GRU-Δt.
• For PhysioNet sepsis prediction, observational intensity improves AUC with Neural CDE achieving 0.880 ± 0.006 AUC with 244 MB memory (OI) versus 0.776–0.833 MB for others (depends on model and setup).
• On Speech Commands with regular sampling, Neural CDE achieves 89.8% ± 2.5% accuracy, while several RNN/ODE baselines show lower performance and higher variance.
• Neural CDEs are theoretically universal approximators for sequences and can represent certain pathwise functions, and they subsume other ODE models where the vector field depends on the continuous input.