[Paper Review] Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Yiping Lu, Aoxiao Zhong+2 more|arXiv (Cornell University)|Oct 27, 2017

Model Reduction and Neural Networks48 references153 citations

TL;DR

The paper links popular deep network architectures to numerical discretizations of ODEs, introduces a linear multi-step LM-architecture applied to ResNet/ResNeXt, and shows improved performance with potential parameter compression; it also interprets stochastic training as a stochastic dynamic system.

ABSTRACT

In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress ($>50$\%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

Motivation & Objective

• Motivate design principles for deep nets by linking architectures to differential equation discretizations.
• Introduce the LM-architecture and apply it to ResNet/ResNeXt to improve accuracy with fewer parameters.
• Explain performance gains via modified equations from numerical analysis.
• Explore stochastic training as approximating stochastic dynamic systems to enhance generalization.

Proposed method

• Map ResNet/ResNeXt and related networks to numerical schemes solving u_t = f(u) (forward Euler, backward Euler, Runge-Kutta).
• Propose LM-architecture: u_{n+1} = (1 - k_n) u_n + k_n u_{n-1} + f(u_n) with trainable k_n.
• Apply LM-architecture to ResNet/ResNeXt to form LM-ResNet/LM-ResNeXt and evaluate on CIFAR and ImageNet.
• Analyze modified equations to explain performance gains and stability.
• Describe stochastic learning strategies (noise injections) as approximations to stochastic dynamics and extend to LM-architecture (stochastic depth).

Experimental results

Research questions

• RQ1 Can deep network architectures be interpreted as discretizations of differential equations, and does this interpretation guide architecture design?
• RQ2 Does the LM-architecture improve performance and/or parameter efficiency for ResNet/ResNeXt on CIFAR and ImageNet?
• RQ3 How do modified equations explain the observed gains from LM-architecture?
• RQ4 Can stochastic training strategies be understood as stochastic dynamic systems and benefit LM-architectures?

Key findings

• LM-ResNet/LM-ResNeXt achieve higher accuracy than their ResNet/ResNeXt counterparts with similar parameter counts on CIFAR and ImageNet.
• On CIFAR, LM-ResNet/LM-ResNeXt can significantly compress the original networks (>50%) while maintaining similar performance.
• On CIFAR-10/ CIFAR-100, LM-ResNet/LM-ResNeXt show notable improvements over baseline architectures across varying depths.
• On ImageNet, LM-ResNet50/ResNet50 and LM-ResNet101/ResNet101 demonstrate improved top-1/top-5 accuracy with comparable parameter budgets (e.g., LM-ResNet50 top-1 23.8 vs ResNet50 24.7; top-1 22.6 vs 23.6 for LM-ResNet101 vs ResNet101).
• Stochastic depth and other noise injections can further improve performance, and can be naturally incorporated into LM-architectures via stochastic dynamic system interpretation.
• The modified equation analysis explains how LM-structure coefficients (k_n) influence acceleration and stability of the learned dynamics.