[Paper Review] A Fine-Grained Spectral Perspective on Neural Networks

Greg Yang, Hadi Salman|arXiv (Cornell University)|Jul 24, 2019

Gaussian Processes and Bayesian Inference59 references42 citations

TL;DR

The paper studies Conjugate Kernel (CK) and Neural Tangent Kernel (NTK) spectra to understand initialization, training dynamics, and hyperparameter effects, revealing non-universal simplicity bias, depth trade-offs, and learning-rate predictions.

ABSTRACT

Are neural networks biased toward simple functions? Does depth always help learn more complex features? Is training the last layer of a network as good as training all layers? How to set the range for learning rate tuning? These questions seem unrelated at face value, but in this work we give all of them a common treatment from the spectral perspective. We will study the spectra of the *Conjugate Kernel, CK,* (also called the *Neural Network-Gaussian Process Kernel*), and the *Neural Tangent Kernel, NTK*. Roughly, the CK and the NTK tell us respectively "what a network looks like at initialization" and "what a network looks like during and after training." Their spectra then encode valuable information about the initial distribution and the training and generalization properties of neural networks. By analyzing the eigenvalues, we lend novel insights into the questions put forth at the beginning, and we verify these insights by extensive experiments of neural networks. We derive fast algorithms for computing the spectra of CK and NTK when the data is uniformly distributed over the boolean cube, and show this spectra is the same in high dimensions when data is drawn from isotropic Gaussian or uniformly over the sphere. Code replicating our results is available at github.com/thegregyang/NNspectra.

Motivation & Objective

• Investigate whether neural networks favor simple functions via spectral properties of CK and NTK.
• Analyze how hyperparameters (depth, variance, activation) shape the spectrum and implications for learning and generalization.
• Examine the trade-offs between training only the last layer versus training all layers in terms of feature complexity.
• Predict maximal non-diverging learning rates using spectral theory and validate across distributions and real data.

Proposed method

• Define CK and NTK for infinitely wide MLPs and show their inductive kernel recurrences.
• Show CK/NTK act as kernels of Gaussian processes and linearized training dynamics, enabling spectral analysis.
• Diagonalize CK/NTK on the boolean cube using Fourier basis to obtain eigenvalues by degree.
• Introduce fractional variance as a heuristic to relate spectral content to learnability of functions.
• Empirically verify spectral predictions on boolean cube, sphere, Gaussian distributions and real datasets (MNIST, CIFAR10).
• Develop and use fast computations of CK/NTK spectra for boolean cube and high-dimensional settings.

Experimental results

Research questions

• RQ1 Do CK and NTK spectra reveal a universal simplicity bias across architectures and depths?
• RQ2 How do depth, activation, and weight/bias variances influence the top eigenvalues and degree-wise fractional variance?
• RQ3 When is training only the last layer preferable to training all layers for learning features of varying complexity?
• RQ4 Can spectral properties predict practical learning-rate limits (max learning rate) for SGD across data distributions and datasets?

Key findings

• Simplicity bias is not universal; for sigmoidal networks the bias can be weakened or eliminated by depth and weight variance, though a weak form persists in eigenvalue ordering by even/odd degrees.
• Deeper networks can learn more complex features, but there exists an optimal depth beyond which performance degrades for higher-degree components.
• NTK generally assigns more variance to higher-degree components than CK, favoring learning of more complex features when training all layers.
• For a given degree, optimal depth to maximize fractional variance increases with degree, indicating depth helps more for higher-complexity features but only up to a point.
• The framework yields accurate qualitative and, to a degree, quantitative predictions of maximal SGD learning rates across distributions and real data, aligning with empirical results.
• Empirical results on MNIST and CIFAR10 corroborate the boolean-cube and high-dimensional theory in predicting spectral and training behavior.