[Paper Review] Sharp Global Guarantees for Nonconvex Low-rank Recovery in the Noisy Overparameterized Regime

Richard Y. Zhang|arXiv (Cornell University)|Apr 21, 2021

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper establishes sharp global convergence guarantees for nonconvex low-rank matrix recovery in the overparameterized regime, where the search rank $ r $ exceeds the true rank $ r^\star $. It proves that no spurious local minima exist when the restricted isometry constant $ \delta < 1/(1 + \sqrt{r^\star/r}) $, and this bound is tight—counterexamples exist when $ \delta \geq 1/(1 + 1/\sqrt{r - r^\star + 1}}) $, showing the bound cannot be improved without additional assumptions on $ r^\star $. The results are sharp for rank-1 and full-rank cases, with improved thresholds when $ r^\star $ is known a priori.

ABSTRACT

Recent work established that rank overparameterization eliminates spurious local minima in nonconvex low-rank matrix recovery under the restricted isometry property (RIP). But this does not fully explain the practical success of overparameterization, because real algorithms can still become trapped at nonstrict saddle points (approximate second-order points with arbitrarily small negative curvature) even when all local minima are global. Moreover, the result does not accommodate for noisy measurements, but it is unclear whether such an extension is even possible, in view of the many discontinuous and unintuitive behaviors already known for the overparameterized regime. In this paper, we introduce a novel proof technique that unifies, simplifies, and strengthens two previously competing approaches -- one based on escape directions and the other based on the inexistence of counterexample -- to provide sharp global guarantees in the noisy overparameterized regime. We show, once local minima have been converted into global minima through slight overparameterization, that near-second-order points achieve the same minimax-optimal recovery bounds (up to small constant factors) as significantly more expensive convex approaches. Our results are sharp with respect to the noise level and the solution accuracy, and hold for both the symmetric parameterization $XX^{T}$, as well as the asymmetric parameterization $UV^{T}$ under a balancing regularizer; we demonstrate that the balancing regularizer is indeed necessary.

Motivation & Objective

• To provide sharp global convergence guarantees for nonconvex low-rank matrix recovery when the search rank $ r $ exceeds the true rank $ r^\star $.
• To characterize the exact threshold of the restricted isometry property (RIP) constant $ \delta $ that ensures no spurious local minima exist.
• To show that the RIP threshold depends critically on $ r^\star $, with tighter bounds when $ r^\star $ is known in advance.
• To establish that the bound $ \delta < 1/2 $ is both necessary and sufficient for exact recovery when $ r^\star = r $, but can be improved when $ r^\star < r $.

Proposed method

• Derives a sharp sufficient condition for the absence of spurious local minima using the restricted isometry property (RIP) with $ \delta < 1/(1 + \sqrt{r^\star/r}) $, under $ (\delta, r + r^\star) $-RIP.
• Constructs explicit counterexamples to show that the bound is tight, proving necessity of $ \delta \geq 1/(1 + 1/\sqrt{r - r^\star + 1}}) $ for the existence of spurious minima.
• Uses convex relaxation techniques to reformulate the problem of finding the minimal $ \delta $ for which a point is a second-order critical point, enabling exact characterization.
• Applies variational analysis and matrix perturbation theory to derive lower bounds on the RIP threshold via optimization over $ X, Z $ with $ \mathrm{rank}(Z) = r^\star $.
• Extends the analysis to approximate second-order optimality, introducing $ \delta(X,Z,\epsilon_g,\epsilon_H) $ to quantify robustness under gradient and Hessian inexactness.
• Leverages the Eckart–Young theorem and low-rank approximation duality to connect the optimization problem to classical matrix approximation theory.

Experimental results

Research questions

• RQ1 What is the tightest possible RIP constant $ \delta $ that guarantees no spurious local minima in nonconvex low-rank matrix recovery when $ r^\star < r $?
• RQ2 How does the true rank $ r^\star $ affect the RIP threshold for exact recovery in the overparameterized regime?
• RQ3 Can the RIP threshold be improved when $ r^\star $ is known a priori, such as in the case $ r^\star = 1 $?
• RQ4 Is the bound $ \delta < 1/2 $ both necessary and sufficient for exact recovery when $ r^\star = r $, and why is this the sharpest possible without prior knowledge of $ r^\star $?
• RQ5 What is the role of overparameterization in enabling tighter RIP-based guarantees, and how does it differ from the standard $ r^\star = r $ case?

Key findings

• The sufficient condition $ \delta < 1/(1 + \sqrt{r^\star/r}) $ guarantees no spurious local minima in the overparameterized regime $ r^\star \leq r $, and this bound is sharp.
• When $ r^\star = r $, the necessary and sufficient RIP threshold is $ \delta < 1/2 $, which is the tightest possible without prior knowledge of $ r^\star $.
• For rank-1 ground truths ($ r^\star = 1 $), the RIP threshold improves to $ \delta < 1/(1 + 1/\sqrt{r}) $, allowing a larger condition number for the measurement operator.
• The necessity of the bound $ \delta \geq 1/(1 + 1/\sqrt{r - r^\star + 1}}) $ is demonstrated via explicit counterexamples, proving the bound cannot be improved.
• The analysis reveals that higher true rank $ r^\star $ leads to stricter RIP requirements, while lower $ r^\star $ allows looser, more robust thresholds.
• The results generalize to approximate second-order optimality, showing that small inexactness in gradient and Hessian leads to a controlled 'loss' in the RIP threshold, quantified by $ \epsilon_g\|y\| + \epsilon_H\mathrm{tr}(W) $.