[論文レビュー] No bad local minima: Data independent training error guarantees for multilayer neural networks

Daniel Soudry, Yair Carmon|arXiv (Cornell University)|May 26, 2016

Stochastic Gradient Optimization Techniques参考文献 22被引用数 159

ひとこと要約

本論文は平滑化分析を用いて、穏やかな過剰パラメータ化の下で、分岐線的活性化関数と二次損失を用いる多層ニューラルネットワークの微分可能な局所最小値すべてが訓練誤差ゼロになることを示す。まず1つの隠れ層の場合、次により深いネットワークへ拡張する。

ABSTRACT

We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with one hidden layer, the training error is zero at every differentiable local minimum, for almost every dataset and dropout-like noise realization. We then extend these results to the case of more than one hidden layer. Our theoretical guarantees assume essentially nothing on the training data, and are verified numerically. These results suggest why the highly non-convex loss of such MNNs can be easily optimized using local updates (e.g., stochastic gradient descent), as observed empirically.

研究の動機と目的

• Motivate why SGD succeeds in training non-convex MNN losses despite potential bad local minima.
• Provide data-independent training error guarantees under mild over-parameterization.
• Demonstrate zero training error at differentiable local minima for networks with one hidden layer and extend to deeper architectures.

提案手法

• Model MNNs with piecewise linear activations and dropout-like noise to enable smoothed analysis.
• Derive gradient conditions at differentiable local minima and formulate a gradient matrix G whose rank controls zero training error.
• Prove that if the last hidden layer has enough parameters (N ≤ d_{L-2}d_{L-1}), then differentiable local minima yield zero training error with probability 1 over data and dropout realizations.
• For L=2 (one hidden layer), show that rank(G1)=N almost everywhere when N ≤ d0 d1.
• For L≥3, show that perturbing last two layers and fixing earlier layers yields global minima with zero training error under N ≤ d_{L-2}d_{L-1}.

実験結果

リサーチクエスチョン

• RQ1 Under mild over-parameterization, can zero training error be guaranteed at differentiable local minima for MNNs with piecewise linear activations?
• RQ2 How does network depth affect the existence of zero-training-error differentiable local minima under a smoothed-analysis framework?
• RQ3 Can dropout-like noise and data perturbations render all differentiable local minima globally optimal in training error?
• RQ4 What role does the rank of the gradient matrix play in ensuring zero training error at local minima?

主な発見

• For a single hidden layer, if N ≤ d0 d1, all differentiable local minima have zero training error almost everywhere.
• Extending to multiple hidden layers, if N ≤ dL-2 dL-1, perturbing the last two layers (with earlier layers fixed) yields global minima with zero training error almost everywhere.
• The results hold with respect to Lebesgue measure over data and dropout realizations, implying data-independence of the guarantees.
• Dropout-like noise ensures the gradient matrix G_{L-1} has full column rank, which under mild over-parameterization leads to zero training error at DLMs.
• The Hessian at differentiable local minima is positive semidefinite, and the zero-error condition becomes typical rather than pathological under random perturbations.
• Numerical experiments on synthetic and MNIST-derived datasets show training error approaching zero in over-parameterized regimes.