[Paper Review] Neural Controlled Differential Equations for Online Prediction Tasks

James Morrill, Patrick Kidger+2 more|arXiv (Cornell University)|Jun 21, 2021

Machine Learning in Healthcare29 references10 citations

TL;DR

This paper proposes rectilinear and cubic Hermite spline control paths for Neural Controlled Differential Equations (Neural CDEs), enabling true online prediction by ensuring causality and measurability. The authors demonstrate state-of-the-art performance on three MIMIC-IV ICU monitoring tasks, outperforming both ODE benchmarks and SOTA non-ODE methods in online settings.

ABSTRACT

Neural controlled differential equations (Neural CDEs) are a continuous-time extension of recurrent neural networks (RNNs), achieving state-of-the-art (SOTA) performance at modelling functions of irregular time series. In order to interpret discrete data in continuous time, current implementations rely on non-causal interpolations of the data. This is fine when the whole time series is observed in advance, but means that Neural CDEs are not suitable for use in extit{online prediction tasks}, where predictions need to be made in real-time: a major use case for recurrent networks. Here, we show how this limitation may be rectified. First, we identify several theoretical conditions that interpolation schemes for Neural CDEs should satisfy, such as boundedness and uniqueness. Second, we use these to motivate the introduction of new schemes that address these conditions, offering in particular measurability (for online prediction), and smoothness (for speed). Third, we empirically benchmark our online Neural CDE model on three continuous monitoring tasks from the MIMIC-IV medical database: we demonstrate improved performance on all tasks against ODE benchmarks, and on two of the three tasks against SOTA non-ODE benchmarks.

Motivation & Objective

• To address the limitation of existing Neural CDEs in online prediction tasks due to non-causal data interpolation.
• To formalize theoretical conditions—boundedness, uniqueness, measurability, and smoothness—that ideal control schemes for Neural CDEs should satisfy.
• To develop new interpolation schemes that satisfy these conditions, particularly enabling real-time inference.
• To empirically validate the proposed schemes on continuous monitoring tasks from the MIMIC-IV database.
• To provide practical recommendations for selecting control path schemes based on task requirements (online/offline, missing data, speed vs. accuracy).

Proposed method

• Propose rectilinear control paths as a causally consistent, measurable interpolation scheme suitable for online prediction.
• Introduce cubic Hermite splines with backward differences to improve smoothness and speed while maintaining online feasibility.
• Formalize theoretical conditions for control paths, including boundedness, uniqueness, and reparameterization invariance.
• Implement the new schemes in the torchcde open-source library for broad accessibility.
• Use the continuous adjoint method for efficient training and inference in continuous time.
• Benchmark the online Neural CDE with rectilinear and Hermite control paths against ODE and SOTA non-ODE baselines on MIMIC-IV tasks.

Experimental results

Research questions

• RQ1 What theoretical properties must a control path for Neural CDEs satisfy to enable online prediction?
• RQ2 How can interpolation schemes for Neural CDEs be designed to be causally consistent and measurable in real-time?
• RQ3 What trade-offs exist between speed, accuracy, and online feasibility in control path design for Neural CDEs?
• RQ4 How do rectilinear and cubic Hermite spline control paths compare empirically to existing schemes (e.g., linear, natural cubic splines) in online and offline settings?
• RQ5 Which control path scheme offers the best performance-accuracy-speed trade-off for online ICU monitoring tasks?

Key findings

• The rectilinear control path is the only scheme that enables truly continuous online prediction, as it depends only on past and current data.
• Cubic Hermite splines with backward differences offer a strong balance of speed and performance, outperforming linear and rectilinear schemes in speed while maintaining competitive accuracy.
• The proposed online Neural CDE model achieves state-of-the-art performance on two of three MIMIC-IV continuous monitoring tasks, outperforming SOTA non-ODE benchmarks.
• Natural cubic splines are the fastest among all schemes tested, making them ideal for offline tasks where speed is prioritized.
• Empirical results show that rectilinear and Hermite-based schemes consistently outperform linear and natural cubic spline baselines in online settings on ICU monitoring tasks.
• The implementation of rectilinear and Hermite control paths in the torchcde library enables broader adoption and reproducibility of the proposed methods.