[Paper Review] Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach

Ryo Karakida, Shotaro Akaho+1 more|arXiv (Cornell University)|Jun 4, 2018

Neural Networks and Applications46 references50 citations

TL;DR

The paper derives universal asymptotic statistics of the Fisher information matrix (FIM) for deep networks with random weights and large width, showing many eigenvalues near zero and a few very large ones, and links these to learning strategies.

ABSTRACT

The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which enables us to utilize mean field theories. We investigate the asymptotic statistics of the FIM's eigenvalues and reveal that most of them are close to zero while the maximum eigenvalue takes a huge value. Because the landscape of the parameter space is defined by the FIM, it is locally flat in most dimensions, but strongly distorted in others. Moreover, we demonstrate the potential usage of the derived statistics in learning strategies. First, small eigenvalues that induce flatness can be connected to a norm-based capacity measure of generalization ability. Second, the maximum eigenvalue that induces the distortion enables us to quantitatively estimate an appropriately sized learning rate for gradient methods to converge.

Motivation & Objective

• Motivate a universal geometric view of parameter space across wide, randomly initialized DNNs using the Fisher information matrix (FIM).
• Derive asymptotic statistics of FIM eigenvalues (mean, variance, maximum) in large-width limits via mean field theory.
• Show how macroscopic, layer-wise variables govern FIM statistics through recursive relations.
• Explore practical implications for learning, including Fisher-Rao norm interpretation and learning-rate guidance for gradient methods.

Proposed method

• Adopt random connectivity with large width and fixed layer-wise width ratios to enable mean field analysis.
• Define and compute macroscopic order parameters (hat{q}^l, hat{q}_{st}^l, tilde{q}^l, tilde{q}_{st}^l) through forward/backpropagation recursions.
• Use the dual matrix F^* to derive eigenvalue statistics of the FIM in the large M limit.
• Establish Theorems 1–4 giving mean, second moment, and maximum eigenvalue in terms of macroscopic variables.
• Relate FIM statistics to learning strategies via Theorem 5 (Fisher-Rao norm) and Theorem 7 (critical learning rate).

Experimental results

Research questions

• RQ1 Do FIM eigenvalues of deep networks with random weights exhibit universal asymptotic statistics across architectures?
• RQ2 How do the mean, variance, and maximum eigenvalues of the FIM scale with network width and depth?
• RQ3 Can macroscopic mean-field variables fully determine FIM statistics through layer-wise recursions?
• RQ4 How can FIM-derived statistics inform generalization capacity measures and practical learning rates?
• RQ5 What are the implications of FIM geometry for optimization methods such as natural gradients or momentum-based SGD?

Key findings

• The mean of the FIM eigenvalues scales as m_{lambda} = C kappa_{1} / M, vanishing as O(1/M).
• The second moment s_{lambda} is O(1), indicating a skew with many near-zero eigenvalues but a heavy tail.
• The maximum eigenvalue scales as lambda_max = alpha ((T-1)/T kappa_{2} + (1/T) kappa_{1}) M, indicating a large, dominant direction.
• Fisher-Rao norm can be bounded and is largely governed by kappa_{1} in large-width limits (Theorem 5).
• A learning-rate bound emerges: eta_c = 2(1+mu) / ( alpha ((T-1)/T kappa_{2} + (1/T) kappa_{1}) M ), guiding convergence (Theorem 7).
• The theory aligns with experiments across tanh, ReLU, and linear activations for large M.