[Paper Review] The Emergence of Spectral Universality in Deep Networks

Jeffrey Pennington, Samuel S. Schoenholz+1 more|arXiv (Cornell University)|Feb 27, 2018

Blind Source Separation TechniquesComputer Science5 references61 citations

TL;DR

The paper develops a free-probability based framework to characterize the full spectrum of the input-output Jacobian in deep networks at initialization, revealing universal limiting spectral distributions that persist with depth for various nonlinearities and weight types.

ABSTRACT

Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spectra of Jacobians at initialization. To this end, we leverage powerful tools from free probability theory to provide a detailed analytic understanding of how a deep network's Jacobian spectrum depends on various hyperparameters including the nonlinearity, the weight and bias distributions, and the depth. For a variety of nonlinearities, our work reveals the emergence of new universal limiting spectral distributions that remain concentrated around one even as the depth goes to infinity.

Motivation & Objective

• Understand how the entire Jacobian spectrum depends on nonlinearity, weight, bias distributions, and depth.
• Characterize the limiting spectral distribution of the Jacobian for deep networks at initialization.
• Identify regimes ensuring well-conditioned Jacobians and dynamical isometry for fast learning.
• Develop a tractable master equation to compute Jacobian spectra from nonlinearities and weight transforms.

Proposed method

• Model the Jacobian J as a product of Diagonal(D^l) and weight matrices W^l across L layers.
• Use free probability, specifically S-transforms, to derive an implicit master equation for the spectrum of J J^T.
• Express the master equation in terms of the moment generating function of the nonlinearity M_{D^2} and the S-transform of weights S_{W^T W}.
• Specialize to common nonlinearities (e.g., linear, ReLU, Hard Tanh, Erf) and weight ensembles (orthogonal, Gaussian) to obtain universal spectral predictions.
• Numerically extract spectra via solving the master equations and a root-finding procedure for G(z).
• Compute moments m_k of the Jacobian spectrum from expansions of M_{D^2} and S_{W^T W} to assess variance scaling with depth.

Experimental results

Research questions

• RQ1 What is the full limiting spectral distribution of the input-output Jacobian J J^T in deep networks at initialization?
• RQ2 How do nonlinearities, weight/bias distributions, and network depth influence the spectrum beyond its mean (chi^L)?
• RQ3 Do universal limiting spectral distributions emerge as depth grows, and under what conditions (e.g., orthogonal weights, critical initialization)?
• RQ4 Can a compact master equation capture the spectrum for arbitrary nonlinearities and weight transforms?

Key findings

• The entire spectrum of J J^T concentrates around one under critical initialization, and its moments depend on depth via chi^L, but on the critical boundary the mean remains one while higher moments grow with depth.
• Two universal limiting spectral classes emerge: Bernoulli-like (Hard Tanh, Shifted ReLU) and Smooth (Erf, Smoothed ReLU) for orthogonal weights, with distinct limiting distributions.
• ReLU destroys the stable limiting spectrum, whereas Hard Tanh and Erf can yield depth-invariant variance in certain double-scaling limits.
• A master equation relates M_{D^2} and S_{W^T W} to the spectrum, enabling practical computation of the full density via G(z) and the Stieltjes inversion.
• Spectral edges and delta-function peaks in the limiting density can be predicted from the analytic form of G(z) and the Lambert W function in Bernoulli class.
• For orthogonal networks, the universality classes persist across depth, with convergence rates dependent on the nonlinearity.