[Paper Review] First-order Methods Almost Always Avoid Saddle Points

Jason D. Lee, Ioannis Panageas+4 more|arXiv (Cornell University)|Oct 20, 2017

Stochastic Gradient Optimization TechniquesComputer Science12 references76 citations

TL;DR

This work proves that many first-order optimization methods avoid strict saddle points for almost all initializations, without second-order info or extra randomness beyond initialization.

ABSTRACT

We establish that first-order methods avoid saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical systems perspective in which appropriate instantiations of the Stable Manifold Theorem allow for a global stability analysis. Thus, neither access to second-order derivative information nor randomness beyond initialization is necessary to provably avoid saddle points.

Motivation & Objective

• Motivate non-convex optimization by the prevalence of saddle points and their impact on first-order methods.
• Generalize prior gradient-descent analyses to a broad class of first-order algorithms.
• Provide a unified framework to prove saddle-point avoidance without relying on stochastic perturbations or Hessian-based methods.

Proposed method

• Model first-order algorithms as dynamical systems via maps g and analyze their fixed points.
• Use the Stable Manifold Theorem to show the global stable set of strict saddles has measure zero.
• Establish invertibility of the Jacobian (det(Dg(x)) ≠ 0) under mild smoothness assumptions to apply measure-zero results.
• Apply the main theorem to gradient descent, proximal point, coordinate descent, block coordinate descent, and mirror descent.
• Demonstrate that strict saddle points are unstable fixed points for these methods.

Experimental results

Research questions

• RQ1 Do first-order methods avoid strict saddle points for almost all initializations under mild regularity conditions?
• RQ2 Can a wide class of algorithms (gradient descent, proximal point, coordinate descent, etc.) be analyzed via dynamical systems to establish saddle-point avoidance?
• RQ3 Is invertibility of the update map (det(Dg(x)) ≠ 0) sufficient to apply a Stable Manifold-type measure-zero result for attractors at saddles?
• RQ4 Does this avoidance hold without adding stochastic noise or second-order information?
• RQ5 What are the implications for convergence to local minima across standard non-convex settings?

Key findings

• First-order methods avoid strict saddle points for almost all initializations under mild regularity conditions.
• The result applies to gradient descent, proximal point, block coordinate descent, coordinate descent, and mirror descent.
• A dynamical-systems approach with the Stable Manifold Theorem yields that the stable set of unstable fixed points has measure zero.
• Under Lipschitz gradient assumptions, strict saddles are unstable fixed points of gradient descent and related methods.
• Det(Dg(x)) ≠ 0 holds under the identified assumptions, enabling the measure-zero argument.
• The framework provides a unified theory for why classic first-order heuristics converge to local minima in non-convex landscapes.