[Paper Review] Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry

Yossi Arjevani, Michael Field|arXiv (Cornell University)|Aug 4, 2020

Physics of Superconductivity and Magnetism37 references4 citations

TL;DR

This paper provides the first rigorous analytic characterization of the Hessian spectrum in shallow ReLU networks using symmetry and representation theory. It shows that for $ d o k $, $ dk - O(d) $ eigenvalues concentrate near zero, while $ \Omega(d) $ eigenvalues grow linearly with $ k $, resolving long-standing empirical observations about extreme spectral skewness in a provable way.

ABSTRACT

We consider the optimization problem associated with fitting two-layers ReLU networks with respect to the squared loss, where labels are generated by a target network. We leverage the rich symmetry structure to analytically characterize the Hessian at various families of spurious minima in the natural regime where the number of inputs $d$ and the number of hidden neurons $k$ is finite. In particular, we prove that for $d\ge k$ standard Gaussian inputs: (a) of the $dk$ eigenvalues of the Hessian, $dk - O(d)$ concentrate near zero, (b) $Ω(d)$ of the eigenvalues grow linearly with $k$. Although this phenomenon of extremely skewed spectrum has been observed many times before, to our knowledge, this is the first time it has been established {rigorously}. Our analytic approach uses techniques, new to the field, from symmetry breaking and representation theory, and carries important implications for our ability to argue about statistical generalization through local curvature.

Motivation & Objective

• To analytically characterize the Hessian spectrum at critical points in two-layer ReLU networks with finite $ d $ and $ k $.
• To resolve the long-observed phenomenon of extreme spectral skewness in the Hessian—where most eigenvalues are near zero and a few are large—through rigorous analysis.
• To develop a novel framework based on group symmetry and representation theory to decompose the Hessian at spurious minima.
• To test and validate longstanding hypotheses in deep learning about curvature, optimization, and generalization using exact spectral results.
• To establish the existence and spectral properties of type A, I, and II spurious minima for $ d \geq k \geq 6 $.

Proposed method

• Leverage the invariance of the loss under row and column permutations of the weight matrix $ \mathbf{W} $, corresponding to $ S_k \times S_d $ symmetry.
• Use representation theory to decompose the Hessian into isotypic components, enabling exact spectral analysis at critical points.
• Apply power series expansions in $ 1/\sqrt{k} $ to describe critical points and their Hessian structure.
• Derive explicit expressions for Hessian blocks using geometric quantities like angles $ \theta_{\mathbf{w}_i, \mathbf{w}_j} $ and norms of weight vectors.
• Use the $ S_k $-representation of the Hessian to compute eigenvalues via irreducible representations, particularly $ H_{k-1} $ and $ T $, for symmetric critical points.
• Validate results numerically by perturbing trained models and observing eigenvalue clustering around predicted values.

Experimental results

Research questions

• RQ1 What is the analytic structure of the Hessian spectrum at spurious minima in shallow ReLU networks with finite $ d $ and $ k $?
• RQ2 Why do empirical studies consistently observe a highly skewed Hessian spectrum with most eigenvalues near zero and a few large ones?
• RQ3 Can the symmetry of the student-teacher ReLU model be used to derive exact spectral results for the Hessian at critical points?
• RQ4 Do type A, I, and II spurious minima exist for $ d \geq k \geq 6 $, and what are their spectral properties?
• RQ5 How do the spectral properties of global minima and spurious minima compare, and what does this imply for the flat minima hypothesis?

Key findings

• For $ d \geq k $, $ dk - O(d) $ eigenvalues of the Hessian concentrate near zero, indicating a massive null space.
• At least $ \Omega(d) $ eigenvalues grow linearly with $ k $, confirming the presence of a few large curvature directions.
• Type II spurious minima exhibit $ k+1 $ outlier eigenvalues: $ k $ of them scale as $ k/4 $, and one as $ k/(2\pi) $.
• Type A spurious minima have $ k $ eigenvalues growing as $ k/4 $, and one as $ (k+1)/4 $, with small corrections of order $ O(k^{-1/2}) $.
• The Hessian spectrum of global minima and type A spurious minima is nearly indistinguishable for $ k=50 $, challenging the flat minima conjecture.
• The Hessian at $ \widetilde{\mathbf{W}} = \widetilde{\mathbf{V}} $ has eigenvalues $ 1/4 $ (multiplicity $ m(k-1) $) and $ (k+2)/4 $ (multiplicity $ m $), derived from $ S_k $-representation decomposition.