[Paper Review] Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition

Rong Ge, Furong Huang+2 more|arXiv (Cornell University)|Mar 6, 2015

Tensor decomposition and applications27 references611 citations

TL;DR

This paper introduces a strict saddle property for non-convex optimization, proving that stochastic gradient descent (SGD) with noise can efficiently escape saddle points and converge to a local minimum in polynomial time. The authors apply this framework to orthogonal tensor decomposition, proposing a new objective function with the strict saddle property, enabling the first online algorithm with global convergence guarantees for tensor decomposition.

ABSTRACT

We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allows for efficient optimization. Using this property we show that stochastic gradient descent converges to a local minimum in a polynomial number of iterations. To the best of our knowledge this is the first work that gives global convergence guarantees for stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points. Our analysis can be applied to orthogonal tensor decomposition, which is widely used in learning a rich class of latent variable models. We propose a new optimization formulation for the tensor decomposition problem that has strict saddle property. As a result we get the first online algorithm for orthogonal tensor decomposition with global convergence guarantee.

Motivation & Objective

• To identify conditions under which stochastic gradient descent (SGD) can efficiently escape saddle points in non-convex optimization.
• To address the challenge of saddle points being a major bottleneck in training deep neural networks and other non-convex models.
• To provide global convergence guarantees for SGD in non-convex settings with exponentially many local minima and saddle points.
• To develop an online algorithm for orthogonal tensor decomposition with theoretical convergence guarantees.

Proposed method

• Introduce the strict saddle property: a twice-differentiable function where every saddle point has a Hessian with at least one negative eigenvalue.
• Analyze noisy gradient descent (SGD with injected noise) under the strict saddle condition, proving convergence to a local minimum in polynomial time.
• Design a new optimization formulation for orthogonal tensor decomposition that satisfies the strict saddle property.
• Use Riemannian optimization tools, including tangent space projections and Hessian approximations, to analyze local geometry around critical points.
• Prove that under the strict saddle condition, SGD escapes saddle points efficiently due to stochastic noise, even with only first-order gradient information.
• Establish global convergence for the proposed online algorithm via stability and curvature analysis of the Hessian on the constraint manifold.

Experimental results

Research questions

• RQ1 Under what conditions can stochastic gradient descent escape saddle points efficiently in non-convex optimization?
• RQ2 Can first-order methods like SGD achieve global convergence in non-convex problems with exponentially many saddle points and local minima?
• RQ3 Does the strict saddle property—where all saddle points have a negative curvature direction—enable efficient escape from saddle points with only gradient information?
• RQ4 Can the strict saddle property be engineered into practical problems like orthogonal tensor decomposition?
• RQ5 Is there a globally convergent online algorithm for orthogonal tensor decomposition?

Key findings

• Stochastic gradient descent with noise converges to a local minimum in polynomial time under the strict saddle property, even in the presence of exponentially many saddle points.
• The proposed optimization formulation for orthogonal tensor decomposition satisfies the strict saddle property, enabling global convergence guarantees.
• The first online algorithm for orthogonal tensor decomposition is proposed with global convergence, overcoming scalability limitations of batch methods.
• The analysis shows that saddle points are unstable under SGD due to noise, allowing escape even when gradient is zero.
• For any point within a δ-neighborhood of a local minimum, the Hessian has positive curvature in all tangent directions, ensuring stability and convergence.
• The framework applies to both unconstrained and equality-constrained optimization, extending to problems like tensor decomposition with orthogonality constraints.