[Paper Review] No bad local minima: Data independent training error guarantees for multilayer neural networks

Daniel Soudry, Yair Carmon|arXiv (Cornell University)|May 26, 2016

Stochastic Gradient Optimization Techniques22 references159 citations

TL;DR

The paper uses smoothed analysis to prove that, under mild over-parameterization, all differentiable local minima of multilayer neural networks with piecewise linear activations and quadratic loss have zero training error, first for one hidden layer and then extended to deeper networks.

ABSTRACT

We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with one hidden layer, the training error is zero at every differentiable local minimum, for almost every dataset and dropout-like noise realization. We then extend these results to the case of more than one hidden layer. Our theoretical guarantees assume essentially nothing on the training data, and are verified numerically. These results suggest why the highly non-convex loss of such MNNs can be easily optimized using local updates (e.g., stochastic gradient descent), as observed empirically.

Motivation & Objective

• Motivate why SGD succeeds in training non-convex MNN losses despite potential bad local minima.
• Provide data-independent training error guarantees under mild over-parameterization.
• Demonstrate zero training error at differentiable local minima for networks with one hidden layer and extend to deeper architectures.

Proposed method

• Model MNNs with piecewise linear activations and dropout-like noise to enable smoothed analysis.
• Derive gradient conditions at differentiable local minima and formulate a gradient matrix G whose rank controls zero training error.
• Prove that if the last hidden layer has enough parameters (N ≤ d_{L-2}d_{L-1}), then differentiable local minima yield zero training error with probability 1 over data and dropout realizations.
• For L=2 (one hidden layer), show that rank(G1)=N almost everywhere when N ≤ d0 d1.
• For L≥3, show that perturbing last two layers and fixing earlier layers yields global minima with zero training error under N ≤ d_{L-2}d_{L-1}.

Experimental results

Research questions

• RQ1 Under mild over-parameterization, can zero training error be guaranteed at differentiable local minima for MNNs with piecewise linear activations?
• RQ2 How does network depth affect the existence of zero-training-error differentiable local minima under a smoothed-analysis framework?
• RQ3 Can dropout-like noise and data perturbations render all differentiable local minima globally optimal in training error?
• RQ4 What role does the rank of the gradient matrix play in ensuring zero training error at local minima?

Key findings

• For a single hidden layer, if N ≤ d0 d1, all differentiable local minima have zero training error almost everywhere.
• Extending to multiple hidden layers, if N ≤ dL-2 dL-1, perturbing the last two layers (with earlier layers fixed) yields global minima with zero training error almost everywhere.
• The results hold with respect to Lebesgue measure over data and dropout realizations, implying data-independence of the guarantees.
• Dropout-like noise ensures the gradient matrix G_{L-1} has full column rank, which under mild over-parameterization leads to zero training error at DLMs.
• The Hessian at differentiable local minima is positive semidefinite, and the zero-error condition becomes typical rather than pathological under random perturbations.
• Numerical experiments on synthetic and MNIST-derived datasets show training error approaching zero in over-parameterized regimes.