[Paper Review] Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Over-parameterization

Jianhao Ma, Salar Fattahi|arXiv (Cornell University)|Feb 17, 2022

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper establishes global convergence of the sub-gradient method (SubGM) for robust low-rank matrix recovery under noisy, corrupted measurements and over-parameterized models. It proves that small initialization nullifies the negative effects of over-parameterization and noise, enabling SubGM to converge exponentially fast to the true low-rank solution—even with arbitrarily large and dense noise—via a novel Sign-Restricted Isometry Property (Sign-RIP).

ABSTRACT

In this work, we study the performance of sub-gradient method (SubGM) on a natural nonconvex and nonsmooth formulation of low-rank matrix recovery with $\ell_1$-loss, where the goal is to recover a low-rank matrix from a limited number of measurements, a subset of which may be grossly corrupted with noise. We study a scenario where the rank of the true solution is unknown and over-estimated instead. The over-estimation of the rank gives rise to an over-parameterized model in which there are more degrees of freedom than needed. Such over-parameterization may lead to overfitting, or adversely affect the performance of the algorithm. We prove that a simple SubGM with small initialization is agnostic to both over-parameterization and noise in the measurements. In particular, we show that small initialization nullifies the effect of over-parameterization on the performance of SubGM, leading to an exponential improvement in its convergence rate. Moreover, we provide the first unifying framework for analyzing the behavior of SubGM under both outlier and Gaussian noise models, showing that SubGM converges to the true solution, even under arbitrarily large and arbitrarily dense noise values, and--perhaps surprisingly--even if the globally optimal solutions do not correspond to the ground truth. At the core of our results is a robust variant of restricted isometry property, called Sign-RIP, which controls the deviation of the sub-differential of the $\ell_1$-loss from that of an ideal, expected loss. As a byproduct of our results, we consider a subclass of robust low-rank matrix recovery with Gaussian measurements, and show that the number of required samples to guarantee the global convergence of SubGM is independent of the over-parameterized rank.

Motivation & Objective

• To analyze the global convergence of the sub-gradient method (SubGM) for nonconvex, nonsmooth low-rank matrix recovery with ℓ₁-loss.
• To address the challenge of over-parameterization, where the search rank r′ exceeds the true rank r, which typically degrades convergence and induces overfitting.
• To study the impact of noisy and grossly corrupted measurements on SubGM performance, particularly under arbitrary noise levels.
• To unify the analysis of SubGM under both outlier and Gaussian noise models, showing robustness to extreme noise.
• To demonstrate that small initialization renders SubGM agnostic to over-parameterization and noise, enabling near-optimal convergence rates.

Proposed method

• Proposes a signal-residual decomposition of the SubGM solution trajectory, separating the low-rank (signal) and residual components.
• Introduces a novel robust variant of the restricted isometry property, called Sign-RIP, to control sub-differential deviation from ideal loss behavior.
• Employs a three-phase analysis: initial transient phase, intermediate phase with residual decay, and final exponential convergence phase.
• Uses one-step dynamics of signal, cross, and residual terms (via Propositions 7–9) to bound the evolution of the residual norm and signal error.
• Establishes that small initialization ensures the residual term remains bounded and decays, enabling the signal term to converge exponentially.
• Leverages the fact that small initialization keeps the sub-gradient step size small and the residual component negligible, effectively simulating a lower-rank model.

Experimental results

Research questions

• RQ1 Can SubGM achieve global convergence for robust low-rank matrix recovery under arbitrary noise levels, including gross corruptions?
• RQ2 Does small initialization eliminate the negative impact of over-parameterization on SubGM convergence, even when r′ ≫ r?
• RQ3 Can SubGM converge to the true solution even when global or local minima do not correspond to the ground truth?
• RQ4 Is the convergence rate of SubGM independent of the over-parameterized rank r′ under Gaussian measurements?
• RQ5 How does the proposed Sign-RIP condition control the behavior of the sub-differential in non-smooth, non-convex settings?

Key findings

• SubGM with small initialization converges globally to the true low-rank solution at an exponential rate, regardless of over-parameterization.
• Small initialization nullifies the effect of over-parameterization, effectively making SubGM behave as if r′ = r, leading to exponential convergence improvement.
• SubGM converges to the ground truth even under arbitrarily large and dense noise, including cases where the globally optimal solution does not match the true matrix.
• The required number of samples for global convergence of SubGM is independent of the over-parameterized rank r′ in the Gaussian measurement model.
• The proposed Sign-RIP condition ensures that the sub-differential of the ℓ₁-loss remains close to that of an ideal loss, enabling convergence analysis.
• The residual term in the signal-residual decomposition remains bounded and decays over time, while the signal term converges exponentially fast to the true solution.