[Paper Review] Elimination of All Bad Local Minima in Deep Learning

Kenji Kawaguchi, Leslie Pack Kaelbling|arXiv (Cornell University)|Jan 2, 2019

Sparse and Compressive Sensing Techniques32 references38 citations

TL;DR

The authors prove that adding one neuron per output unit eliminates all suboptimal local minima for any deep network under practical loss assumptions, with the added neurons’ effects vanishing at local minima, and introduce a PGB-based analysis.

ABSTRACT

In this paper, we theoretically prove that adding one special neuron per output unit eliminates all suboptimal local minima of any deep neural network, for multi-class classification, binary classification, and regression with an arbitrary loss function, under practical assumptions. At every local minimum of any deep neural network with these added neurons, the set of parameters of the original neural network (without added neurons) is guaranteed to be a global minimum of the original neural network. The effects of the added neurons are proven to automatically vanish at every local minimum. Moreover, we provide a novel theoretical characterization of a failure mode of eliminating suboptimal local minima via an additional theorem and several examples. This paper also introduces a novel proof technique based on the perturbable gradient basis (PGB) necessary condition of local minima, which provides new insight into the elimination of local minima and is applicable to analyze various models and transformations of objective functions beyond the elimination of local minima.

Motivation & Objective

• Motivate the challenge of poor local minima in deep nets and seek theoretical guarantees beyond over-parameterization.
• Propose a minimal architectural modification that eliminates all suboptimal local minima.
• Prove that at any local minimum, the original network parameters achieve a global minimum of the base objective.
• Characterize failure modes of local minima elimination and introduce a new proof technique (PGB) applicable beyond this problem.

Proposed method

• Define an auxiliary objective by adding one neuron per output unit with an exponential form and a regularizer on the added weights.
• Show that local minima of the auxiliary objective imply the original network is globally optimal for the base objective.
• Use the perturbable gradient basis (PGB) necessary condition to derive the elimination result.
• Provide proofs under standard differentiable and convex loss criteria and discuss weaker realizable-data assumptions.
• Characterize a failure mode and discuss implications for gradient-based optimization.

Experimental results

Research questions

• RQ1 Can a single, minimal architectural change guarantee elimination of all bad local minima in deep networks across tasks and losses?
• RQ2 Under what assumptions does this elimination hold, and how does the added neuron interact with the original network parameters at optimum?
• RQ3 What are the limitations or failure modes of this elimination approach, particularly for gradient-based optimization?
• RQ4 How does the new PGB framework illuminate broader transformations of objective functions beyond local minima elimination.

Key findings

• Adding one neuron per output unit ensures that every local minimum of the modified objective corresponds to a global minimum of the original objective.
• At every local minimum, the modified network reduces to the original network, and the added components vanish.
• The results hold for multi-class classification, binary classification, and regression with common loss functions under mild assumptions.
• A novel PGB necessary condition provides a general tool to analyze local minima and is applicable beyond this specific problem.