[Paper Review] Training Very Deep Networks

Rupesh K. Srivastava, Klaus Greff+1 more|arXiv (Cornell University)|Jul 22, 2015

Advanced Neural Network Applications23 references1,100 citations

TL;DR

This paper introduces highway networks, a deep residual architecture with learnable gating units that enable unimpeded information flow across hundreds of layers. By using adaptive gating mechanisms inspired by LSTM, the model allows direct training via standard stochastic gradient descent, overcoming vanishing gradient problems in very deep networks.

ABSTRACT

Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.

Motivation & Objective

• To address the challenge of training very deep neural networks due to vanishing gradients and optimization difficulties.
• To design a network architecture that enables stable and efficient training of networks with hundreds of layers.
• To introduce a mechanism for unimpeded information flow across layers through adaptive gating.
• To enable direct training of extremely deep architectures using standard backpropagation without residual shortcuts or skip connections.

Proposed method

• Introduces highway networks with learnable gating units that control the flow of information between layers.
• Uses a gating mechanism similar to LSTM, where a gate determines how much of the input is passed through unchanged.
• Defines the transformation as a combination of a linear transformation and a gated identity skip connection: H(x) = x * T(x) + H(x) * (1 - T(x)), where T(x) is the transform gate.
• Employs standard backpropagation and stochastic gradient descent for end-to-end training without architectural modifications.
• Applies the highway unit across multiple stacked layers, enabling deep architectures with residual-like behavior.

Experimental results

Research questions

• RQ1 Can very deep neural networks (hundreds of layers) be trained effectively using standard optimization methods?
• RQ2 How can information flow be preserved across hundreds of layers to prevent vanishing gradients?
• RQ3 Can adaptive gating mechanisms replace or outperform residual connections in enabling deep network training?
• RQ4 What is the performance and stability of highway networks compared to standard deep networks on benchmark tasks?

Key findings

• Highway networks successfully train networks with over 100 layers using standard stochastic gradient descent without residual shortcuts.
• The architecture achieves stable training dynamics even at extreme depths, demonstrating robustness to vanishing gradients.
• The model maintains high performance on image classification tasks, showing that deep architectures can be trained effectively with the proposed gating mechanism.
• The use of adaptive gates enables efficient information flow, allowing the network to learn complex representations without degradation.