[Paper Review] Spurious Valleys in Two-layer Neural Network Optimization Landscapes

Luca Venturi, Afonso S. Bandeira+1 more|arXiv (Cornell University)|Feb 18, 2018

Stochastic Gradient Optimization TechniquesComputer Science48 references50 citations

TL;DR

The paper defines intrinsic dimensions for one-hidden-layer networks and shows that finite intrinsic dimension prevents spurious valleys in over-parameterized regimes, while infinite intrinsic dimension allows them; spurious valleys, when present, reside at low risk and are less likely as width grows.

ABSTRACT

Neural networks provide a rich class of high-dimensional, non-convex optimization problems. Despite their non-convexity, gradient-descent methods often successfully optimize these models. This has motivated a recent spur in research attempting to characterize properties of their loss surface that may explain such success. In this paper, we address this phenomenon by studying a key topological property of the loss: the presence or absence of spurious valleys, defined as connected components of sub-level sets that do not include a global minimum. Focusing on a class of two-layer neural networks defined by smooth (but generally non-linear) activation functions, we identify a notion of intrinsic dimension and show that it provides necessary and sufficient conditions for the absence of spurious valleys. More concretely, finite intrinsic dimension guarantees that for sufficiently overparametrised models no spurious valleys exist, independently of the data distribution. Conversely, infinite intrinsic dimension implies that spurious valleys do exist for certain data distributions, independently of model overparametrisation. Besides these positive and negative results, we show that, although spurious valleys may exist in general, they are confined to low risk levels and avoided with high probability on overparametrised models.

Motivation & Objective

• Motivate understanding of non-convex loss landscapes in neural networks.
• Characterize the presence or absence of spurious valleys in one-hidden-layer networks.
• Introduce intrinsic dimension concepts to relate architecture to optimization topologies.
• Establish conditions under which overparameterization eliminates spurious valleys for various activations.
• Contrast empirical risk and population risk in the context of landscape properties.

Proposed method

• Define spurious valleys as connected components of sub-level sets not containing a global minimum.
• Introduce upper and lower intrinsic dimensions to quantify functional space complexity of networks.
• Prove that finite intrinsic dimension guarantees absence of spurious valleys for sufficiently wide networks.
• Show that infinite intrinsic dimension implies existence of spurious valleys for some data distributions.
• Provide specialized results for polynomial activations and ERM settings via corollaries.
• Discuss improvements for linear and quadratic activations and relate to tensor decompositions.

Experimental results

Research questions

• RQ1 When do spurious valleys exist or vanish in one-hidden-layer neural networks?
• RQ2 How do intrinsic dimensions of the network affect the optimization landscape under overparameterization?
• RQ3 Do results differ between population risk and empirical risk minimization across activation types?
• RQ4 Can overparameterization guarantee valley-free optimization for specific activation classes (e.g., polynomial, linear, quadratic)?

Key findings

• Spurious valleys do not occur when the hidden width p is at least the upper intrinsic dimension dim*(σ,X) for finite dim*(σ,X).
• For polynomial activations, no spurious valleys arise in ERM and population risk under sufficient over-parameterization; results are tight up to constants for linear/quadratic activations.
• For non-polynomial, non-negative activations, spurious valleys can exist for any width by constructing adversarial data distributions.
• Overparameterized models can still exhibit spurious valleys, but their measure decreases with width, and low-energy spurious valleys are avoided with high probability.
• In linear networks, no spurious valleys occur for square loss across any depth; for quadratic activations with m=1, p ≥ 2n+1 suffices to avoid spurious valleys.