[Paper Review] When Are Nonconvex Problems Not Scary?

Ju Sun, Qing Qu+1 more|arXiv (Cornell University)|Oct 21, 2015

Sparse and Compressive Sensing Techniques44 references112 citations

TL;DR

This paper proposes a second-order trust-region algorithm that provably converges to a global minimizer for a class of nonconvex problems where all local minima are global and all saddle points have negative curvature. The method efficiently escapes saddle points using Hessian-based descent directions, ensuring convergence from any initialization.

ABSTRACT

In this note, we focus on smooth nonconvex optimization problems that obey: (1) all local minimizers are also global; and (2) around any saddle point or local maximizer, the objective has a negative directional curvature. Concrete applications such as dictionary learning, generalized phase retrieval, and orthogonal tensor decomposition are known to induce such structures. We describe a second-order trust-region algorithm that provably converges to a global minimizer efficiently, without special initializations. Finally we highlight alternatives, and open problems in this direction.

Motivation & Objective

• To identify a broad class of nonconvex optimization problems that are tractable despite NP-hardness in general.
• To explain why heuristic algorithms like gradient descent often succeed in practice for problems such as dictionary learning and phase retrieval.
• To develop a provably convergent algorithm that escapes saddle points and local maxima in nonconvex problems with specific geometric structure.
• To establish conditions under which all local minimizers are global, and saddle points have negative curvature, enabling efficient global optimization.

Proposed method

• Proposes a second-order trust-region algorithm that uses a quadratic approximation of the objective function around each iterate using Riemannian Hessian and gradient information.
• Defines the Riemannian trust-region subproblem by minimizing the quadratic model within a trust region of radius Δ in the tangent space of the manifold.
• Employs retraction maps to project the updated search direction back onto the manifold, ensuring iterates remain feasible.
• Leverages negative curvature in the Hessian at saddle points and local maximizers to identify descent directions that escape these points.
• Uses local approximation accuracy and ridability parameters to ensure sufficient decrease in objective value at each step.
• Establishes convergence to a global minimizer by showing that descent steps are always available at non-optimal points, with quadratic convergence near the solution.

Experimental results

Research questions

• RQ1 Under what conditions on the objective function can nonconvex problems be solved efficiently despite NP-hardness in general?
• RQ2 Why do gradient-based heuristics often succeed in practice for nonconvex problems like dictionary learning and phase retrieval?
• RQ3 Can a trust-region algorithm provably escape saddle points and local maxima when the Hessian has at least one negative eigenvalue at such points?
• RQ4 Is it possible to design a globally convergent algorithm for nonconvex problems where all local minima are global and all saddle points are 'ridable'?
• RQ5 What are the minimal assumptions on the objective function and manifold structure to ensure global convergence of second-order methods?

Key findings

• The proposed trust-region algorithm converges to a global minimizer for all (α,β,γ,δ)-X functions, a class of nonconvex problems where all local minimizers are global and all saddle points have negative curvature.
• The algorithm guarantees sufficient decrease in the objective at each iteration by exploiting negative curvature directions in the Hessian, enabling escape from saddle points and local maxima.
• Convergence is guaranteed from any initialization, eliminating the need for careful or problem-specific initialization strategies.
• Near the global minimizer, the algorithm exhibits quadratic convergence when the trust region is unconstrained, resembling Newton’s method.
• The method is robust to local approximation errors as long as the trust region radius Δ is sufficiently small, ensuring reliable descent.
• Empirical and theoretical results suggest that problems such as dictionary learning, generalized phase retrieval, and orthogonal tensor decomposition fall into this favorable class.