[Paper Review] Random Walks: Training Very Deep Nonlinear Feed-Forward Networks with Smart Initialization.

David Sussillo|arXiv (Cornell University)|Dec 19, 2014

Stochastic Gradient Optimization Techniques27 citations

TL;DR

This paper proposes a smart initialization scheme for training very deep feed-forward networks by modeling gradient norm evolution as a random walk. By scaling weight matrices to balance the random walk of the log-gradient norm, the method ensures stable training with gradient norms scaling as the square root of depth, mitigating vanishing gradients through increased width rather than depth.

ABSTRACT

Training very deep networks is an important open problem in machine learning. One of many difficulties is that the norm of the back-propagated error gradient can grow or decay exponentially. Here we show that training very deep feed-forward networks (FFNs) is not as difficult as previously thought. Unlike when back-propagation is applied to a recurrent network, application to an FFN amounts to multiplying the error gradient by a different random matrix at each layer. We show that the successive application of correctly scaled random matrices to an initial vector results in a random walk of the log of the norm of the resulting vectors, and we compute the scaling that makes this walk unbiased. The variance of the random walk grows only linearly with network depth and is inversely proportional to the size of each layer. Practically, this implies a gradient whose log-norm scales with the square root of the network depth and shows that the vanishing gradient problem can be mitigated by increasing the width of the layers. Mathematical analyses and experimental results using stochastic gradient descent to optimize tasks related to the MNIST and TIMIT datasets are provided to support these claims. Equations for the optimal matrix scaling are provided for the linear and ReLU cases.

Motivation & Objective

• To address the challenge of vanishing or exploding gradients in very deep feed-forward networks.
• To understand how gradient norm evolves during back-propagation in deep FFNs with random weight matrices.
• To derive optimal weight scaling that stabilizes gradient flow by making the log-norm of gradients perform an unbiased random walk.
• To show that increasing layer width, not depth, is key to mitigating vanishing gradients.

Proposed method

• Models the back-propagated gradient as a sequence of multiplications by random weight matrices, each scaled to control variance.
• Analyzes the log-norm of the gradient vector as a random walk, deriving conditions for an unbiased walk.
• Computes optimal scaling factors for weight matrices in both linear and ReLU-activated networks to stabilize the random walk.
• Uses the variance of the random walk, which grows linearly with depth and inversely with layer width, to guide initialization.
• Derives analytical expressions for optimal scaling in the linear and ReLU cases based on the random walk's properties.
• Validates the approach using stochastic gradient descent on MNIST and TIMIT tasks with empirically measured gradient norms.

Experimental results

Research questions

• RQ1 How does the norm of the back-propagated gradient evolve in very deep feed-forward networks with random weights?
• RQ2 What scaling of weight matrices ensures that the log-norm of the gradient performs an unbiased random walk?
• RQ3 How does the variance of the gradient norm's random walk depend on network depth and layer width?
• RQ4 Can increasing layer width stabilize training in very deep networks by controlling gradient variance?
• RQ5 What are the optimal initialization scales for linear and ReLU-activated deep networks to prevent vanishing or exploding gradients?

Key findings

• The log-norm of the back-propagated gradient performs a random walk whose variance grows linearly with network depth.
• The optimal weight scaling ensures the random walk is unbiased, preventing systematic decay or explosion of the gradient norm.
• The variance of the random walk is inversely proportional to the width of each layer, meaning wider layers stabilize gradient flow.
• As a result, the expected magnitude of the gradient scales with the square root of the network depth, not exponentially.
• Empirical results on MNIST and TIMIT show that the proposed initialization enables stable training of very deep networks.
• The derived scaling rules for linear and ReLU networks are shown to be effective in practice, with gradient norms remaining stable across hundreds of layers.