[Paper Review] Tensor Programs III: Neural Matrix Laws

Greg Yang|arXiv (Cornell University)|Sep 22, 2020

Stochastic Gradient Optimization TechniquesComputer Science50 references19 citations

TL;DR

This paper introduces the Free Independence Principle (FIP), proving that in randomly initialized wide neural networks, pre-activations become asymptotically free from weight matrices in the sense of random matrix theory. Using a new Master Theorem for Tensor Programs, the authors rigorously justify the free independence assumption used in prior work on Jacobian singular value distributions and the Neural Tangent Kernel, establishing architectural universality across all neural network architectures.

ABSTRACT

In a neural network (NN), *weight matrices* linearly transform inputs into *preactivations* that are then transformed nonlinearly into *activations*. A typical NN interleaves multitudes of such linear and nonlinear transforms to express complex functions. Thus, the (pre-)activations depend on the weights in an intricate manner. We show that, surprisingly, (pre-)activations of a randomly initialized NN become *independent* from the weights as the NN's widths tend to infinity, in the sense of asymptotic freeness in random matrix theory. We call this the Free Independence Principle (FIP), which has these consequences: 1) It rigorously justifies the calculation of asymptotic Jacobian singular value distribution of an NN in Pennington et al. [36,37], essential for training ultra-deep NNs [48]. 2) It gives a new justification of gradient independence assumption used for calculating the Neural Tangent Kernel of a neural network. FIP and these results hold for any neural architecture. We show FIP by proving a Master Theorem for any Tensor Program, as introduced in Yang [50,51], generalizing the Master Theorems proved in those works. As warmup demonstrations of this new Master Theorem, we give new proofs of the semicircle and Marchenko-Pastur laws, which benchmarks our framework against these fundamental mathematical results.

Motivation & Objective

• To establish a rigorous foundation for the free independence assumption widely used in analyzing neural network Jacobian singular value distributions.
• To generalize random matrix theory techniques to nonlinear deep learning settings where classical methods fail.
• To prove that the free independence property holds universally across all neural network architectures, not just specific ones.
• To develop a new Master Theorem for Tensor Programs that enables systematic analysis of asymptotic behavior in wide neural networks.
• To provide new, rigorous proofs of classical random matrix laws (semicircle and Marchenko-Pastur) as benchmarks for the proposed framework.

Proposed method

• Proposes a new Master Theorem for Tensor Programs that generalizes earlier results, enabling asymptotic analysis of complex nonlinear compositions in neural networks.
• Uses a recursive induction framework on vector sets (M) to track moment convergence and orthogonality properties under matrix transformations.
• Applies pseudo-Lipschitz functions and Gaussian conditioning techniques to handle nonlinear dependencies in weight matrices and pre-activations.
• Employs a decomposition of matrix products into Gaussian and linear components to prove vanishing moments for residual terms.
• Introduces the concept of 'vanishing moments' and 'orthogonal systems' to formalize asymptotic independence in high-dimensional limits.
• Leverages strong law of large numbers and concentration bounds to establish almost sure convergence of moment statistics.

Experimental results

Research questions

• RQ1 Does the free independence assumption between weights and pre-activations hold universally across all neural network architectures in the infinite-width limit?
• RQ2 Can classical random matrix laws like the semicircle and Marchenko-Pastur laws be rigorously re-derived using a unified framework for nonlinear neural networks?
• RQ3 Is the expansion technique from classical random matrix theory effective for analyzing nonlinear neural network Jacobians?
• RQ4 What mathematical conditions ensure that pre-activations become asymptotically free from weight matrices in wide networks?
• RQ5 Can a general Master Theorem be formulated for Tensor Programs that captures the asymptotic behavior of arbitrary neural network architectures?

Key findings

• The Free Independence Principle (FIP) is rigorously proven: in randomly initialized wide neural networks, pre-activations become asymptotically free from weight matrices in the sense of free probability theory.
• The FIP justifies the free independence assumption used in prior calculations of Jacobian singular value distributions, which are critical for achieving dynamical isometry in ultra-deep networks.
• The proposed Master Theorem for Tensor Programs successfully recovers the semicircle and Marchenko-Pastur laws as special cases, validating the framework against fundamental results in random matrix theory.
• The framework establishes architectural universality: the FIP holds for any neural network architecture, regardless of depth, width, or activation function.
• The method proves that residual terms arising from nonlinear transformations (e.g., ReLU, tanh) have vanishing moments in the infinite-width limit, enabling asymptotic independence.
• The analysis shows that matrix products involving nonlinearly dependent matrices (e.g., Jacobians of deep networks) can be decomposed into components with vanishing moments, enabling rigorous asymptotic analysis.