[Paper Review] The critical locus of overparameterized neural networks

Yaim Cooper|arXiv (Cornell University)|May 8, 2020

Stochastic Gradient Optimization Techniques17 references4 citations

TL;DR

This paper investigates the geometry of critical points in overparameterized feedforward neural networks, identifying a positive-dimensional 'star locus' S and a degenerate core locus C within it. For deep networks (depth ≥ 4), it proves that critical points in C are degenerate with exactly one nonzero Hessian eigenvalue, and shows that the number of zero Hessian eigenvalues grows linearly with width m, providing foundational insights into why gradient descent may converge to non-global minima.

ABSTRACT

Many aspects of the geometry of loss functions in deep learning remain mysterious. In this paper, we work toward a better understanding of the geometry of the loss function $L$ of overparameterized feedforward neural networks. In this setting, we identify several components of the critical locus of $L$ and study their geometric properties. For networks of depth $\ell \geq 4$, we identify a locus of critical points we call the star locus $S$. Within $S$ we identify a positive-dimensional sublocus $C$ with the property that for $p \in C$, $p$ is a degenerate critical point, and no existing theoretical result guarantees that gradient descent will not converge to $p$. For very wide networks, we build on earlier work and show that all critical points of $L$ are degenerate, and give lower bounds on the number of zero eigenvalues of the Hessian at each critical point. For networks that are both deep and very wide, we compare the growth rates of the zero eigenspaces of the Hessian at all the different families of critical points that we identify. The results in this paper provide a starting point to a more quantitative understanding of the properties of various components of the critical locus of $L$.

Motivation & Objective

• To understand the geometric structure of the critical locus of the loss function L in overparameterized deep neural networks.
• To identify and characterize positive-dimensional families of critical points, especially degenerate ones, that could trap gradient descent.
• To quantify how the dimension of critical loci and the number of zero Hessian eigenvalues scale with network width m.
• To compare the growth rates of different critical point families in deep and very wide networks.
• To provide theoretical bounds on the degeneracy of critical points, especially in the context of global minima and non-global critical points.

Proposed method

• Identifies the star locus S as a positive-dimensional critical locus in deep networks (depth ℓ ≥ 4), derived from structural symmetries in overparameterized architectures.
• Defines the core locus C ⊂ S as a sublocus of degenerate critical points with exactly one nonzero Hessian eigenvalue.
• Uses algebraic geometry and Hessian analysis to compute the dimension of S and C, showing dim(S) = (ℓ−3)m² + (a+1)m and dim(C) = (ℓ−4)m² + (a+1)m.
• Applies results from [Ngu19] and [LDS18] to show that in very wide networks, all critical points are degenerate, with at least (m−n+1)b zero Hessian eigenvalues.
• Derives lower bounds on the number of zero eigenvalues of the Hessian at all critical points, showing growth linear in m for non-global minima.
• Compares the asymptotic growth of critical loci dimensions and zero eigenspace dimensions across different families of critical points as m increases.

Experimental results

Research questions

• RQ1 What are the geometric and topological properties of the critical locus of the loss function in overparameterized deep neural networks?
• RQ2 Can positive-dimensional families of degenerate critical points exist in deep nonlinear networks, and if so, what are their dimensions?
• RQ3 How does the number of zero eigenvalues of the Hessian grow with network width m at different types of critical points?
• RQ4 How do the dimensions of the locus of global minima, the star locus, and other critical loci compare as network width increases?
• RQ5 What constraints do degeneracy and Hessian structure impose on the convergence behavior of gradient descent?

Key findings

• For deep networks (ℓ ≥ 4), the star locus S is a positive-dimensional critical locus of dimension (ℓ−3)m² + (a+1)m.
• The core locus C ⊂ S consists of degenerate critical points with exactly one nonzero Hessian eigenvalue, and has dimension (ℓ−4)m² + (a+1)m.
• In very wide networks, all critical points are degenerate, with at least (m−n+1)b zero eigenvalues of the Hessian.
• The number of zero Hessian eigenvalues grows linearly with m for all critical points that are not global minima.
• The locus of global minima M has dimension d−bn and grows faster with m than the star locus or other linear subspaces in the level sets.
• As m increases, the dimension of the locus of global minima grows fastest, followed by the star locus, then the linear subspaces within level sets of non-global critical points.