[Paper Review] Global Optimality of Local Search for Low Rank Matrix Recovery

Srinadh Bhojanapalli, Behnam Neyshabur+1 more|arXiv (Cornell University)|May 23, 2016

Sparse and Compressive Sensing TechniquesEngineering27 references158 citations

TL;DR

The paper proves there are no spurious local minima for non-convex factorized low-rank matrix recovery under an incoherence RIP condition, and shows SGD from random initialization converges to a global optimum in polynomial time (noiseless and noisy settings).

ABSTRACT

We show that there are no spurious local minima in the non-convex factorized parametrization of low-rank matrix recovery from incoherent linear measurements. With noisy measurements we show all local minima are very close to a global optimum. Together with a curvature bound at saddle points, this yields a polynomial time global convergence guarantee for stochastic gradient descent {\em from random initialization}.

Motivation & Objective

• Motivate and analyze the matrix sensing problem with a rank-constrained, non-convex factorization.
• Establish absence of spurious local minima under incoherence and RIP-like conditions.
• Show that saddle points have negative curvature, enabling polynomial-time convergence of SGD from random starts.
• Extend results to noisy and approximately low-rank settings.
• Compare with convex relaxations and discuss practical implications for initialization and optimization.

Proposed method

• Study the factorized objective f(U) = ||A(UU^T) − y||^2 with rank constraint by U ∈ R^{n×r}.
• Assume measurement operator A satisfies (2r, δ_{2r})-RIP with δ_{2r} < 1/5 (noiseless) or < 1/10 (noisy).
• Characterize local minima using first- and second-order optimality, and align U with the global optimum U* via an orthogonal transformation R.
• Prove absence of spurious local minima: if y = A(X*) with rank(X*) ≤ r, then U U^T = X* at any local minimum (noiseless).
• Demonstrate a strict saddle property: smallest Hessian eigenvalue at non-global critical points is negative, enabling SGD from random initialization to reach a global optimum in polynomial time.
• Extend to approximate low-rank and noisy cases, showing local minima are close to X* or X*_r with bounds dependent on noise and approximation error.

Experimental results

Research questions

• RQ1 Do spurious local minima exist for the non-convex factorized matrix sensing problem under incoherence/RIP conditions?
• RQ2 Can SGD converge to a global optimum from random initialization for rank-constrained non-convex matrix recovery?
• RQ3 How do noisy measurements and approximate low-rankness affect the quality and location of local minima?
• RQ4 What are the precise conditions (RIP constants, rank, noise level) under which the global optimality of local search is guaranteed?
• RQ5 How do these results compare to convex relaxations in terms of sample complexity and required conditions?

Key findings

• Under (2r, δ_{2r})-RIP with δ_{2r} < 1/5 (noiseless) or δ_{2r} < 1/10 (noisy), every local minimum U satisfies U U^T = X* (exact recovery in noiseless case).
• In the noisy case, all local minima are close to a true factorization X* = U*U*^T, with error bounded by a term that scales with noise and measurements.
• All saddle points have a negative curvature direction, enabling escape and allowing polynomial-time convergence of SGD from random initialization to a global optimum (via existing SGD results for strict saddles).
• For approximate low-rank X*, local minima satisfy ||UU^T − X*||_F bounded by a function of the best rank-r approximation error ||X* − X_r*||_F and δ_{2r} · ||X* − X_r*||_* .
• The required RIP conditions and measurement count (O(nr) with Gaussian measurements) are milder or comparable to prior guarantees, matching optimal sample complexity up to constants.
• The results imply that initialization via SVD is not necessary for global convergence, aligning theory with practical local-search methods.