[Paper Review] Depth Creates No Bad Local Minima

Haihao Lu, Kenji Kawaguchi|arXiv (Cornell University)|Feb 27, 2017

Stochastic Gradient Optimization Techniques20 references81 citations

TL;DR

The paper proves that depth alone does not create bad local minima in deep linear networks, and that all local minima are global minima under full-row-rank conditions. It also provides a simpler proof that all local minima of deep linear networks with square loss are global minima and extends to general loss functions under a key theorem.

ABSTRACT

In deep learning, extit{depth}, as well as extit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatly simplify a recently proposed proof to show that all of the local minima of feedforward deep linear neural networks are global minima. Our theoretical results generalize previous results with fewer assumptions, and this analysis provides a method to show similar results beyond square loss in deep linear models.

Motivation & Objective

• Motivate understanding of how depth contributes to non-convex loss surfaces in deep learning.
• Show that depth alone does not create bad local minima in deep linear networks.
• Provide a simpler proof that all local minima of feedforward deep linear networks are global minima under suitable conditions.
• Extend the results beyond Frobenius (square) loss to general loss functions satisfying a key theorem.
• Discuss implications for matrix completion and broader non-convex optimization problems.

Proposed method

• Formulate the training objective for deep linear networks with square loss as L(W)=1/2 ||W_H ... W_1 X - Y||_F^2.
• Relate the deep problem to a shallow rank-constrained problem F(R)=||RX-Y||_F^2 with rank(R) <= d_p.
• Prove that every local minimum of the deep problem corresponds to a local minimum of the shallow problem (Theorem 2.1).
• Show that all local minima of the rank-constrained shallow problem are global minima (Theorem 2.2).
• Conclude that all local minima of the deep linear network are global minima (Theorem 2.3).
• Provide a constructive perturbation-based proof leveraging SVD perturbation results and rank considerations (Lemmas 3.1–3.4 and Theorems 3.1–3.3).

Experimental results

Research questions

• RQ1 Does depth alone (without nonlinearity) introduce bad local minima in deep linear networks?
• RQ2 Under what conditions do local minima of deep linear networks correspond to global minima of an equivalent shallow model?
• RQ3 Can the no-bad-local-minima property extend to general loss functions beyond Frobenius (square) loss?
• RQ4 How can perturbation theory of SVD be used to relate deep and shallow formulations in terms of rank constraints?

Key findings

• Any local minimum of the deep linear network with square loss yields a corresponding local minimum of a shallow rank-constrained model.
• If the shallow rank-constrained model has full-row-rank input X, all local minima are global minima.
• Therefore, all local minima of feedforward deep linear networks with square loss are global minima under full-row-rank conditions for X and Y.
• The results generalize Kawaguchi (2016) with fewer assumptions and extend beyond square loss via Theorem 3.2.
• The approach provides a path to extend no-bad-local-minima results to matrix completion scenarios with high probability.