[Paper Review] Do Residual Neural Networks discretize Neural Ordinary Differential Equations?

Michael E. Sander, Pierre Ablin+1 more|arXiv (Cornell University)|May 29, 2022

Model Reduction and Neural Networks4 citations

TL;DR

This paper investigates whether Residual Neural Networks (ResNets) approximate Neural Ordinary Differential Equations (Neural ODEs) by analyzing the discretization error between ResNet trajectories and their corresponding continuous ODE solutions. It shows that without depth-smoothness assumptions, the error does not vanish with depth; however, gradient descent implicitly regularizes linear ResNets toward a limit Neural ODE at rate $1/N$, and a memory-free adjoint method with Heun’s integration enables stable training of very deep networks with low memory usage.

ABSTRACT

Neural Ordinary Differential Equations (Neural ODEs) are the continuous analog of Residual Neural Networks (ResNets). We investigate whether the discrete dynamics defined by a ResNet are close to the continuous one of a Neural ODE. We first quantify the distance between the ResNet's hidden state trajectory and the solution of its corresponding Neural ODE. Our bound is tight and, on the negative side, does not go to 0 with depth N if the residual functions are not smooth with depth. On the positive side, we show that this smoothness is preserved by gradient descent for a ResNet with linear residual functions and small enough initial loss. It ensures an implicit regularization towards a limit Neural ODE at rate 1 over N, uniformly with depth and optimization time. As a byproduct of our analysis, we consider the use of a memory-free discrete adjoint method to train a ResNet by recovering the activations on the fly through a backward pass of the network, and show that this method theoretically succeeds at large depth if the residual functions are Lipschitz with the input. We then show that Heun's method, a second order ODE integration scheme, allows for better gradient estimation with the adjoint method when the residual functions are smooth with depth. We experimentally validate that our adjoint method succeeds at large depth, and that Heun method needs fewer layers to succeed. We finally use the adjoint method successfully for fine-tuning very deep ResNets without memory consumption in the residual layers.

Motivation & Objective

• To determine whether the discrete dynamics of a ResNet converge to the continuous solution of a corresponding Neural ODE as depth increases.
• To investigate the role of weight initialization and training dynamics in inducing implicit regularization toward a limit Neural ODE.
• To develop and analyze a memory-free adjoint method for training deep ResNets by reconstructing activations during backpropagation.
• To evaluate the impact of residual function smoothness with depth on gradient approximation accuracy in the adjoint method.
• To validate experimentally that the proposed adjoint method enables effective fine-tuning of very deep ResNets without memory overhead.

Proposed method

• Proposes a framework to define a Neural ODE corresponding to a given ResNet by scaling the residual function with $1/N$, transforming the ResNet update into a Euler discretization of an ODE.
• Derives a theoretical bound on the distance between the ResNet’s hidden state trajectory and the solution of the corresponding Neural ODE, showing the error does not vanish unless residual functions are smooth with depth.
• Analyzes gradient descent dynamics in deep linear ResNets and proves that trained weights converge uniformly to a Lipschitz function at rate $1/N$, implying implicit regularization toward a limit ODE.
• Introduces a memory-free backward pass using a reverse-time Euler scheme to reconstruct activations on the fly, with error bounded by $O(1/N)$ under Lipschitz assumptions on residual functions.
• Extends the method to use Heun’s second-order ODE integration scheme to improve gradient approximation, with error scaling as $O(1/N)$ times the depth-smoothness of residual functions.
• Employs the adjoint method for fine-tuning very deep ResNets by untying weights in a pretrained, tied-weights model, enabling low-memory training and inference.

Experimental results

Research questions

• RQ1 Does the discrete trajectory of a ResNet converge to the solution of its corresponding Neural ODE as depth $N \to \infty$?
• RQ2 Under what conditions does gradient descent on a deep ResNet implicitly regularize the network toward a limit Neural ODE?
• RQ3 Can a memory-free adjoint method accurately reconstruct gradients in deep ResNets without storing activations?
• RQ4 How does the smoothness of residual functions with respect to depth affect the accuracy of gradient estimation in the adjoint method?
• RQ5 Can the proposed adjoint method successfully fine-tune very deep ResNets without memory consumption in residual layers?

Key findings

• The error between the ResNet trajectory and the solution of its corresponding Neural ODE does not converge to zero with depth $N$ unless the residual functions are smooth with respect to depth.
• For deep linear ResNets with small initial loss, gradient descent ensures that the trained parameters converge uniformly to a Lipschitz function at rate $1/N$, implying implicit regularization toward a limit Neural ODE.
• The memory-free adjoint method with reverse-time Euler reconstruction yields an error in gradient estimation bounded by $O(1/N)$ when residual functions are bounded and Lipschitz continuous with constants independent of $N$.
• Using Heun’s method instead of Euler improves gradient approximation, with error scaling as $O(1/N)$ times the smoothness of residual functions with depth, leading to better training stability.
• Empirical results confirm that the adjoint method enables successful fine-tuning of very deep ResNets (e.g., ResNet-101) on CIFAR-10 and ImageNet with no memory overhead in residual layers.
• Heun’s method requires fewer layers to achieve convergence compared to Euler, validating the theoretical advantage of higher-order integration in the adjoint framework.