[论文解读] Neural Controlled Differential Equations for Irregular Time Series

Patrick Kidger, James Morrill+2 more|arXiv (Cornell University)|May 18, 2020

Model Reduction and Neural Networks参考文献 64被引用 154

一句话总结

引入 Neural CDEs，这是一种适用于不规则且部分观测时间序列的连续时间模型，通过受控微分方程更新，并支持内存高效的伴随反向传播，在 ODE/RNN 基线中达到最优状态的结果。

ABSTRACT

Neural ordinary differential equations are an attractive option for modelling temporal dynamics. However, a fundamental issue is that the solution to an ordinary differential equation is determined by its initial condition, and there is no mechanism for adjusting the trajectory based on subsequent observations. Here, we demonstrate how this may be resolved through the well-understood mathematics of \emph{controlled differential equations}. The resulting \emph{neural controlled differential equation} model is directly applicable to the general setting of partially-observed irregularly-sampled multivariate time series, and (unlike previous work on this problem) it may utilise memory-efficient adjoint-based backpropagation even across observations. We demonstrate that our model achieves state-of-the-art performance against similar (ODE or RNN based) models in empirical studies on a range of datasets. Finally we provide theoretical results demonstrating universal approximation, and that our model subsumes alternative ODE models.

研究动机与目标

• Motivate and address limitations of Neural ODEs when data arrive irregularly or partially observed.
• Propose a Neural CDE framework that uses controlled differential equations driven by a data path X.
• Demonstrate memory-efficient adjoint backpropagation across observations.
• Establish universal approximation properties and compare empirically against ODE/RNN baselines.
• Provide practical guidance on implementation and evaluation for irregular time series tasks.

提出的方法

• Define X as a natural cubic spline path through observed data to drive a CDE.
• Formulate z_t = z_{t0} + ∫_{t0}^{t} f_theta(z_s) dX_s with z_{t0} = zeta_theta(x0, t0).
• Use neural networks f_theta and zeta_theta to learn the vector field and initial state.
• Train via adjoint backpropagation with O(H) memory across observations, yielding overall O(L+H) memory.
• Show that a Neural CDE can be written as z_t = z_{t0} + ∫ g_theta,X(z_s, s) ds when X is differentiable, enabling standard ODE solvers.
• Argue that Neural CDEs subsume GRU-ODE-like models and provide universal approximation results.

实验结果

研究问题

• RQ1 Can a continuous-time model driven by irregular data outperform discrete-time RNN/ODE-based approaches on irregular or partially observed time series?
• RQ2 Does the Neural CDE framework support memory-efficient backpropagation across observations while maintaining competitive accuracy?
• RQ3 Do Neural CDEs provide universal approximation capabilities and subsume alternative ODE-based approaches?
• RQ4 How does the observational intensity channel and partial observation handling affect performance on real-world datasets?

主要发现

• Neural CDEs achieve state-of-the-art accuracy among compared models on irregular/partially observed datasets (CharacterTrajectories, PhysioNet sepsis prediction) and demonstrate superior memory efficiency.
• On CharacterTrajectories with 30%, 50%, and 70% data dropped, Neural CDE attains 98.7% ± 0.8% accuracy, with memory usage about 1.3 MB, outperforming GRU-ODE, GRU-D, ODE-RNN, and GRU-Δt.
• For PhysioNet sepsis prediction, observational intensity improves AUC with Neural CDE achieving 0.880 ± 0.006 AUC with 244 MB memory (OI) versus 0.776–0.833 MB for others (depends on model and setup).
• On Speech Commands with regular sampling, Neural CDE achieves 89.8% ± 2.5% accuracy, while several RNN/ODE baselines show lower performance and higher variance.
• Neural CDEs are theoretically universal approximators for sequences and can represent certain pathwise functions, and they subsume other ODE models where the vector field depends on the continuous input.