[Paper Review] Efficient approaches for escaping higher order saddle points in non-convex optimization

Anima Anandkumar, Rong Ge|arXiv (Cornell University)|Feb 18, 2016

Sparse and Compressive Sensing Techniques18 references49 citations

TL;DR

This paper proposes the first efficient algorithm guaranteed to converge to a third-order local minimum in non-convex optimization, using higher-order derivatives to escape degenerate saddle points where first- and second-order methods fail. It proves that finding a fourth-order local minimum is NP-hard, establishing a fundamental limit on higher-order optimization beyond third order.

ABSTRACT

Local search heuristics for non-convex optimizations are popular in applied machine learning. However, in general it is hard to guarantee that such algorithms even converge to a local minimum, due to the existence of complicated saddle point structures in high dimensions. Many functions have degenerate saddle points such that the first and second order derivatives cannot distinguish them with local optima. In this paper we use higher order derivatives to escape these saddle points: we design the first efficient algorithm guaranteed to converge to a third order local optimum (while existing techniques are at most second order). We also show that it is NP-hard to extend this further to finding fourth order local optima.

Motivation & Objective

• Address the challenge of degenerate saddle points in high-dimensional non-convex optimization, where first- and second-order methods can fail due to singular Hessians.
• Develop an efficient algorithm that guarantees convergence to a third-order local minimum, overcoming limitations of existing second-order methods.
• Characterize the conditions under which a point is a third-order local minimum using higher-order derivatives.
• Demonstrate that extending the approach to fourth-order local minima is computationally infeasible by proving NP-hardness.
• Provide theoretical foundations for higher-order optimality in non-convex problems with symmetric or overparameterized structures.

Proposed method

• Define a p-th order local minimum using the condition f(x) - f(y) ≤ o(||x - y||^p) for nearby y, establishing a formal criterion for higher-order optimality.
• Introduce a new algorithm that uses gradient, Hessian, and third-order derivative information to escape degenerate saddle points.
• Design a convergence analysis based on a potential function that tracks progress toward third-order optimality, ensuring convergence in polynomial time.
• Use a regularization technique by adding a degree-6 term ||x||^6 to transform a degree-4 polynomial into a well-behaved function for hardness reduction.
• Prove NP-hardness of finding fourth-order local minima by reducing the non-negativity problem of degree-4 homogeneous polynomials to the optimization problem.
• Leverage the fact that non-negative degree-4 polynomials have the origin as the only fourth-order local minimum, while negative-definite ones have no such minimum with non-negative value.

Experimental results

Research questions

• RQ1 Can higher-order derivatives be used to design an efficient algorithm that escapes degenerate saddle points where first- and second-order methods fail?
• RQ2 What is the computational complexity of finding a fourth-order local minimum in non-convex optimization?
• RQ3 Under what conditions is a critical point a third-order local minimum, and can this be characterized algorithmically?
• RQ4 Are there natural classes of non-convex functions where third-order optimality is both necessary and sufficient for convergence to a local minimum?
• RQ5 Can the hardness of higher-order optimization be formally established using reductions from known NP-hard problems?

Key findings

• The proposed algorithm is guaranteed to converge to a third-order local minimum in polynomial time, providing a provable escape mechanism for degenerate saddle points.
• The algorithm efficiently identifies points that approximately satisfy the necessary and sufficient conditions for third-order local optimality, including small gradient, nearly positive semidefinite Hessian, and bounded third-order derivatives.
• It is proven that finding a fourth-order local minimum is NP-hard, even for well-behaved functions with bounded derivatives and global minimizers within the unit ball.
• The NP-hardness result is established via a reduction from the non-negativity problem of degree-4 homogeneous polynomials, which is itself known to be NP-hard.
• When a degree-4 polynomial is non-negative, the origin is the only fourth-order local minimum; if negative in some direction, all fourth-order local minima must have negative function values.
• The results show a fundamental computational barrier: while third-order optimality is efficiently attainable, higher-order optimality (fourth and beyond) is intractable in general.