[Paper Review] Exponentially vanishing sub-optimal local minima in multilayer neural networks

Daniel Soudry, Elad Hoffer|arXiv (Cornell University)|Feb 19, 2017

Neural Networks and Applications33 references54 citations

TL;DR

The paper proves that in a one-hidden-layer MNN with piecewise linear units trained by MSE, the volume of differentiable regions containing sub-optimal local minima vanishes exponentially relative to global minima under mild over-parameterization and Gaussian input assumptions.

ABSTRACT

Background: Statistical mechanics results (Dauphin et al. (2014); Choromanska et al. (2015)) suggest that local minima with high error are exponentially rare in high dimensions. However, to prove low error guarantees for Multilayer Neural Networks (MNNs), previous works so far required either a heavily modified MNN model or training method, strong assumptions on the labels (e.g., "near" linear separability), or an unrealistic hidden layer with $Ω\left(N ight)$ units. Results: We examine a MNN with one hidden layer of piecewise linear units, a single output, and a quadratic loss. We prove that, with high probability in the limit of $N ightarrow\infty$ datapoints, the volume of differentiable regions of the empiric loss containing sub-optimal differentiable local minima is exponentially vanishing in comparison with the same volume of global minima, given standard normal input of dimension $d_{0}= ildeΩ\left(\sqrt{N} ight)$, and a more realistic number of $d_{1}= ildeΩ\left(N/d_{0} ight)$ hidden units. We demonstrate our results numerically: for example, $0\%$ binary classification training error on CIFAR with only $N/d_{0}\approx 16$ hidden neurons.

Motivation & Objective

• Motivates understanding why SGD finds low-training-error solutions in over-parameterized MNNs.
• Proposes a realistic MNN setting to analyze the prevalence of sub-optimal local minima.
• Derives probabilistic bounds showing sub-optimal regions are exponentially rarer than global minima.
• Seeks to quantify the role of over-parameterization in reducing sub-optimal minima in practical network sizes.

Proposed method

• Analyzes a 2-layer MNN with a single hidden layer of piecewise linear units and a scalar output.
• Uses mean square error loss and differentiable local minima (DLM) as the focus of the analysis.
• Defines differentiable regions where activation patterns are fixed and relates residual errors to a rank condition (A ∘ X) e = 0.
• Introduces angular volume as the probability measure of parameter regions under random Gaussian initialization.
• Proves upper bounds on the angular volume of sub-optimal DLMs and a lower bound on the angular volume of global minima.
• Establishes a main theorem bounding the ratio of volumes, showing exponential vanishing of sub-optimal regions compared to global minima.

Experimental results

Research questions

• RQ1 Under what conditions do sub-optimal differentiable local minima become exponentially rare in high dimensions?
• RQ2 How does over-parameterization (in terms of hidden layer width and input dimension) influence the volume of regions containing sub-optimal versus global minima?
• RQ3 Can realistic assumptions (Gaussian inputs, mild over-parameterization) yield provable guarantees for low training error without modifying the MNN or training method?

Key findings

• The expected angular volume of sub-optimal DLMs with MCE > ε is exponentially small in N under given assumptions.
• Global minima exist with high probability and have a non-negligible angular volume, ensuring a meaningful comparison with sub-optimal regions.
• The ratio of volumes V(Lε) / V(G) is bounded by exp(-γε N^{3/4} (d1 d0)^{1/4}) and is also ≤ exp(-γε N log N), signaling exponential scarcity of sub-optimal regions.
• Numerical experiments on Gaussian data and real datasets (MNIST, CIFAR, ImageNet) show training errors close to zero with relatively modest parameter counts (approx. N parameters), consistent with the theory.
• Non-differentiable critical points appear numerically rare, and the primary results focus on differentiable local minima.