[Paper Review] A Critical View of Global Optimality in Deep Learning

Chulhee Yun, Suvrit Sra+1 more|arXiv (Cornell University)|Feb 10, 2018

Neural dynamics and brain function9 references21 citations

TL;DR

This paper provides a critical analysis of global optimality in deep learning by analyzing the loss surfaces of deep linear and nonlinear networks. It proves that for deep linear networks, critical points inherit structural properties from linear parameterizations, enabling distinction between global minima and saddle points; for nonlinear networks with ReLU-like activations, it demonstrates the existence of infinitely many suboptimal local minima on practical datasets, and constructs counterexamples showing local minima can be strictly worse than the global minimum.

ABSTRACT

We investigate the loss surface of deep linear and nonlinear neural networks. We show that for deep linear networks with differentiable losses, critical points after the multilinear parameterization inherit the structure of critical points of the underlying loss with linear parameterization. As corollaries we obtain minima are results that subsume most previous results, while showing how to distinguish global minima from saddle points. For nonlinear neural networks, we prove two theorems showing that even for networks with one hidden layer, there can be spurious local minima. Indeed, for piecewise linear nonnegative homogeneous activations (e.g., ReLU), we prove that for almost all practical datasets there exist infinitely many local minima that are not global. We conclude by constructing a counterexample involving other activation functions (e.g., sigmoid, tanh, arctan, etc.), for which there exists a local minimum strictly inferior to the global minimum.

Motivation & Objective

• To investigate the structure of critical points in deep linear and nonlinear neural networks.
• To determine whether global minima can be reliably identified and distinguished from saddle points or suboptimal local minima.
• To examine the prevalence of spurious local minima in practical deep learning setups, especially with common nonlinear activations.
• To construct explicit counterexamples where local minima are strictly inferior to the global minimum.

Proposed method

• Analyzing the loss surface of deep linear networks under multilinear parameterization to relate critical points to those of linear parameterizations.
• Using differential geometry and optimization theory to characterize the structure of critical points in linear networks.
• Proving the existence of infinitely many local minima that are not global for one-hidden-layer networks with piecewise linear nonnegative homogeneous activations (e.g., ReLU).
• Constructing explicit counterexamples using non-polyhedral activation functions (e.g., sigmoid, tanh, arctan) to show local minima can be strictly worse than the global minimum.
• Applying measure-theoretic arguments to show that for almost all practical datasets, such suboptimal minima are prevalent.

Experimental results

Research questions

• RQ1 Can global minima in deep linear networks be distinguished from saddle points using the multilinear parameterization?
• RQ2 Do suboptimal local minima exist in nonlinear deep networks with ReLU-like activations on realistic datasets?
• RQ3 Can local minima in deep networks be strictly worse than the global minimum, and if so, under what conditions?
• RQ4 How does the choice of activation function affect the presence and severity of spurious local minima?

Key findings

• For deep linear networks with differentiable losses, critical points under multilinear parameterization inherit the structure of critical points from the linear parameterization, enabling clearer identification of global minima.
• In one-hidden-layer nonlinear networks with ReLU-like activations, there exist infinitely many local minima that are not global minima for almost all practical datasets.
• For certain nonlinear activation functions such as sigmoid, tanh, and arctan, explicit counterexamples demonstrate that local minima can be strictly inferior to the global minimum.
• The results show that the assumption of global optimality in deep learning training is not guaranteed, even in relatively simple architectures with standard nonlinearities.