[Paper Review] Reversible Architectures for Arbitrarily Deep Residual Neural Networks

Bo Chang, Lili Meng+4 more|arXiv (Cornell University)|Sep 12, 2017

Model Reduction and Neural Networks42 references71 citations

TL;DR

The paper presents three stable and reversible architectures for deep ResNet-like models inspired by ODEs, enabling memory-efficient training of arbitrarily deep networks with competitive accuracy and robustness to limited data.

ABSTRACT

Recently, deep residual networks have been successfully applied in many computer vision and natural language processing tasks, pushing the state-of-the-art performance with deeper and wider architectures. In this work, we interpret deep residual networks as ordinary differential equations (ODEs), which have long been studied in mathematics and physics with rich theoretical and empirical success. From this interpretation, we develop a theoretical framework on stability and reversibility of deep neural networks, and derive three reversible neural network architectures that can go arbitrarily deep in theory. The reversibility property allows a memory-efficient implementation, which does not need to store the activations for most hidden layers. Together with the stability of our architectures, this enables training deeper networks using only modest computational resources. We provide both theoretical analyses and empirical results. Experimental results demonstrate the efficacy of our architectures against several strong baselines on CIFAR-10, CIFAR-100 and STL-10 with superior or on-par state-of-the-art performance. Furthermore, we show our architectures yield superior results when trained using fewer training data.

Motivation & Objective

• Motivate and formalize the connection between deep ResNets and ordinary differential equations to study stability and reversibility.
• Develop three reversible architectures with stable forward propagation that allow training of very deep networks under modest resources.
• Introduce regularization to favor smooth time-dynamics in the learned models.
• Demonstrate empirical effectiveness on CIFAR-10, CIFAR-100, and STL-10, including scenarios with limited training data.

Proposed method

• Interpret ResNets as discretized ODEs and analyze forward stability via the Jacobian eigenvalues to ensure non-positive real parts.
• Propose three reversible architectures: a two-layer Hamiltonian network, a MidPoint network, and a Leapfrog network, each with specific discretizations (Verlet-like, central differences, etc.).
• Ensure reversibility so activations can be reconstructed during backpropagation for memory efficiency.
• Provide a regularization functional that enforces smooth time dynamics of time-varying kernels Kin and promotes generalization.
• Demonstrate memory efficiency and trainability for very deep networks (e.g., 1202 layers) and compare against ResNet and RevNet on standard benchmarks.

Experimental results

Research questions

• RQ1 Can ResNet-type architectures be cast as stable, reversible dynamical systems to allow arbitrarily deep networks?
• RQ2 Do Hamiltonian, MidPoint, and Leapfrog inspired blocks provide stability and reversibility in practice for image classification tasks?
• RQ3 How do these reversible architectures perform relative to ResNet and RevNet on CIFAR-10, CIFAR-100, and STL-10, especially with limited training data?
• RQ4 Does the proposed regularization improve generalization by promoting smooth temporal dynamics of network parameters?
• RQ5 Can extremely deep networks be trained with modest memory, and is the training robust to depth?

Key findings

• Three reversible architectures yield stable forward propagation with imaginary Jacobian eigenvalues, enabling memory-efficient backpropagation.
• All three architectures achieve comparable or on-par accuracy with state-of-the-art baselines on CIFAR-10, CIFAR-100, and STL-10, with STL-10 showing notable gains over baselines.
• The Hamiltonian network and its variants demonstrate robustness and better performance when training data is limited compared to ResNet.
• The 1202-layer Hamiltonian network can be trained with half the parameters of a ResNet of similar depth, while maintaining or exceeding accuracy, illustrating memory and optimization stability advantages.
• STL-10 results show the proposed methods outperform benchmark methods by about ten percentage points in some configurations.
• Experiments on data subsampling indicate the stable architectures generalize better than ResNet when labeled data is scarce.