[Paper Review] The loss surface of deep and wide neural networks

Quynh C. Nguyen, Matthias Hein|arXiv (Cornell University)|Apr 26, 2017

Neural Networks and Applications117 citations

TL;DR

The authors prove that for overspecified (very wide) neural networks, almost all local minima are global minima under mild analytic activation and loss assumptions, extending prior results to deep networks with pyramidal architecture.

ABSTRACT

While the optimization problem behind deep neural networks is highly non-convex, it is frequently observed in practice that training deep networks seems possible without getting stuck in suboptimal points. It has been argued that this is the case as all local minima are close to being globally optimal. We show that this is (almost) true, in fact almost all local minima are globally optimal, for a fully connected network with squared loss and analytic activation function given that the number of hidden units of one layer of the network is larger than the number of training points and the network structure from this layer on is pyramidal.

Motivation & Objective

• Motivate understanding of non-convex optimization in deep learning and explain why training often succeeds in practice.
• Characterize the loss surface of deep, wide neural networks under overspecification.
• Generalize previous results from shallow networks to multi-layer architectures with pyramidal structure.

Proposed method

• Formulate feedforward networks with L layers and analytic activation functions.
• Derive backpropagation relations for gradients (Delta matrices) and weight/bias gradients.
• Introduce assumptions on activations and losses (analytic, monotone, and C^2 loss) and prove key lemmas on layer outputs being real analytic functions of parameters.
• Show that when a hidden layer is very wide (n_k ≥ N−1), the rank condition [F_k, 1_N] = N holds on a measure-one set of parameters.
• Use real analytic function properties to argue that the set where rank conditions fail has measure zero.
• Apply the implicit-function theorem and non-degeneracy arguments to conclude global optimality of critical points under specified conditions.

Experimental results

Research questions

• RQ1 Under what conditions on network width and architecture do local minima become global minima in deep networks?
• RQ2 How do analytic activation and loss functions influence the structure of critical points and the prevalence of globally optimal solutions?
• RQ3 Can results known for shallow networks be extended to multi-layer networks with pyramidal, overspecified structures?
• RQ4 What role does the rank of hidden-layer activations play in ensuring global optimality of critical points?

Key findings

• For overspecified networks, a large class of local minima are globally optimal.
• If a hidden layer has at least N−1 units (n_k ≥ N−1), then under mild assumptions every non-degenerate critical point with full-rank upper layers is globally optimal.
• When training samples are linearly independent, every critical point with full column rank in the weight matrices is a global minimum.
• Almost all critical points become global minima under the stated conditions, as problematic points occur on a measure-zero set due to analyticity.
• Degenerate or low-rank saddle points are unlikely in practice if upper-layer weights are full rank and the pyramid structure is maintained.