[Paper Review] Multi-level Residual Networks from Dynamical Systems View

Bo Chang, Lili Meng+3 more|arXiv (Cornell University)|Oct 27, 2017

Advanced Neural Network Applications24 references68 citations

TL;DR

The paper interprets ResNets as dynamical systems and introduces a multi-level training method that interpolates network depth while halving step size, achieving about 40% training time reduction with competitive accuracy on standard datasets.

ABSTRACT

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet theoretically, such as unraveled view, unrolled iterative estimation and dynamical systems view. In this paper, we adopt the dynamical systems point of view, and analyze the lesioning properties of ResNet both theoretically and experimentally. Based on these analyses, we additionally propose a novel method for accelerating ResNet training. We apply the proposed method to train ResNets and Wide ResNets for three image classification benchmarks, reducing training time by more than 40% with superior or on-par accuracy.

Motivation & Objective

• Motivate and analyze residual networks from a dynamical systems perspective to explain lesioning and robustness properties.
• Propose a practical multi-level training method inspired by multi-grid ideas to accelerate ResNet training.
• Demonstrate the method's effectiveness on ResNet and Wide ResNet architectures across standard image classification datasets.

Proposed method

• Model ResNets as discretized ODEs with implicit step size h and show how residual blocks correspond to time steps in an ODE discretization.
• Perform lesion analysis to explain why removing or permuting blocks has limited impact when residual outputs are small.
• Introduce a multi-level training procedure: start with a shallow network (large h), interpolate by inserting blocks after each existing block, then halve h, repeating across cycles.
• Interpolate weights by copying adjacent block parameters to new blocks to initialize deeper models.
• Use cyclic learning rate schedules within each cycle to maintain training performance.
• Provide theoretical time-saving estimates and empirical results showing ~40% reduction in training time while preserving accuracy.

Experimental results

Research questions

• RQ1 How does the dynamical systems interpretation explain the resilience of ResNets to block removal or reordering?
• RQ2 Can a multi-level interpolation strategy accelerate ResNet training without sacrificing accuracy?
• RQ3 What are the practical training-time savings and accuracy trade-offs when applying multi-level training to ResNets and Wide ResNets on standard benchmarks?

Key findings

• The average residue norm scales inversely with the number of blocks, aligning with the dynamical systems view (G(Y_j) ~ T/d).
• Lesion analyses show removing blocks has negligible effect when residual outputs are small, consistent with residuals acting as feature refiners.
• The proposed multi-level training method yields over 40% reduction in training time on ResNets and Wide ResNets across CIFAR-10, CIFAR-100, and STL-10, with superior or comparable accuracy to the last-cycle model.
• Interpolation-based depth expansion initializes deeper models effectively, enabling cycles with three stages (e.g., 2-2-2 to 4-4-4 to 8-8-8, or 5-5-5 to 20-20-20).
• The method maintains competitive test error while significantly reducing wall-clock time, supported by both ResNet and Wide ResNet experiments.