[Paper Review] Neural Ordinary Differential Equations

Ricky T. Q. Chen, Yulia Rubanova+2 more|arXiv (Cornell University)|Jun 19, 2018

Model Reduction and Neural NetworksPhysics and Astronomy55 references566 citations

TL;DR

The paper introduces Neural ODEs, replacing discrete layers with continuous-time dynamics defined by a neural vector field and trained via adjoint sensitivity through a black-box ODE solver, enabling constant memory backpropagation and flexible computation. It also presents continuous normalizing flows and a latent time-series model built on ODEs.

ABSTRACT

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

Motivation & Objective

• Motivate replacing discrete depth with continuous-time dynamics to improve memory efficiency and adaptivity.
• Propose training via adjoint sensitivity to backpropagate through ODE solvers without accessing solver internals.
• Demonstrate continuous-time normalizing flows for scalable, likelihood-based density estimation.
• Introduce continuous-time, latent ODE models for irregular time-series data.
• Show practical benefits in supervised learning, density estimation, and time-series modeling.

Proposed method

• Parameterize hidden-state dynamics with a neural network f(h(t), t, θ) to define dh/dt = f(h, t, θ).
• Solve the resulting initial value problem with a black-box ODE solver to obtain h(T).
• Backpropagate through the solver using the adjoint sensitivity method, solving a backward-in-time augmented ODE for gradients w.r.t. θ.
• Derive an instantaneous change of variables for log-density under continuous transformations, enabling continuous normalizing flows (CNF).
• Train CNFs via maximum likelihood by solving forward and reverse-time ODEs to compute densities and samples.
• Extend the framework to continuous-time latent variable time-series models, enabling irregular observations and Poisson process likelihoods.

Experimental results

Research questions

• RQ1 Can continuous-depth neural networks trained via adjoint differentiation match discrete-depth networks in performance?
• RQ2 Does replacing layered transformations with continuous dynamics provide memory efficiency and adaptive computation without accuracy loss?
• RQ3 Can continuous normalizing flows offer scalable, tractable likelihood-based density estimation without dimension partitioning?
• RQ4 How can ODE-based latent variable models handle irregular time-series data and event times?
• RQ5 What are the trade-offs between forward solver accuracy and training performance in neural ODEs?

Key findings

• ODE-Nets can match ResNet-like performance on MNIST with similar parameter counts.
• Backpropagation through ODE solvers via the adjoint method occurs with linear cost and low memory, often more efficient than backprop through the forward pass.
• Continuous normalizing flows reduce the computational burden by replacing determinant Jacobians with trace-based changes of variables, enabling wide-flow layers with linear cost.
• CNFs and regular NFs show CNFs offering lower training loss in density estimation tasks and better sample transformations in examples like two circles and two moons datasets.
• Latent ODEs enable accurate reconstruction and extrapolation of irregular time-series data, with superior predictive RMSE compared to RNN baselines.