[Paper Review] Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?

Boris Hanin|arXiv (Cornell University)|Jan 11, 2018

Neural Networks and Applications138 citations

TL;DR

The paper rigorously analyzes gradient fluctuations in randomly initialized fully connected ReLU nets and shows that the variance of input-output Jacobian entries grows exponentially with a architecture-dependent sum of reciprocals of hidden layer widths. It provides finite-width results and annealed/quenched EVGP definitions.

ABSTRACT

We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant beta, given by the sum of the reciprocals of the hidden layer widths. When beta is large, the gradients computed by N at initialization vary wildly. Our approach complements the mean field theory analysis of random networks. From this point of view, we rigorously compute finite width corrections to the statistics of gradients at the edge of chaos.

Motivation & Objective

• Motivates the EVGP problem in deep fully connected ReLU networks and connects training dynamics to architecture.
• Derives finite-depth/width formulas for Jacobian moments and identifies a key architecture statistic.
• Characterizes when EVGP occurs via annealed and quenched senses and links it to layer width distribution.
• Offers guidance for architecture design to mitigate EVGP at initialization.

Proposed method

• Derives exact formulas for joint even moments of input-output Jacobian entries in fully connected ReLU nets with random weights and biases (finite depth/width).
• Shows the empirical variance of gradient squares is exponential in beta = sum of 1/n_j over hidden layers.
• Proves EVGP occurrence depends only on architecture under independent initializations with correct variance (Definition 1).
• Defines annealed and quenched EVGP and proves equivalence: EVGP avoided iff sum_j 1/n_j < ∞.
• Computes finite-width corrections to gradient statistics at the edge of chaos and relates to mean-field theory.

Experimental results

Research questions

• RQ1 What combinations of depths and hidden layer widths lead to EVGP at initialization in ReLU nets?
• RQ2 How do finite width and depth affect the statistics of the input-output Jacobian?
• RQ3 Does the EVGP depend on the exact weight/bias distributions or only on architecture under proper initialization?
• RQ4 What architectural choices minimize gradient variance at initialization?
• RQ5 How do annealed and quenched notions of EVGP relate in finite-width networks?

Key findings

• The empirical variance of the squares of input-output Jacobian entries is exponential in beta = sum_j 1/n_j.
• EVGP occurs iff beta is large, with variance growth tied to architecture rather than weight distributions (under correct variance scaling).
• If sum_j 1/n_j < ∞, the architecture avoids EVGP in both annealed and quenched senses; if not, EVGP is expected.
• For fixed input, E[Z_pq^2] = 1/n_0 and higher moments scale exponentially with sum_j 1/n_j; moments are given by path-based combinatorics (Theorem 3).
• Equality in mean-field type inequalities is achieved when hidden widths are equal, suggesting architectural symmetry as favorable to stability.