[Paper Review] Rank Overspecified Robust Matrix Recovery: Subgradient Method and Exact Recovery

Lijun Ding, Liwei Jiang+3 more|arXiv (Cornell University)|Sep 23, 2021

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes a subgradient method with diminishing stepsizes for robust low-rank matrix recovery from grossly corrupted Gaussian measurements, even when the rank is overspecified. It establishes exact recovery at a sublinear rate under the restricted direction preserving property (RDPP), with automatic acceleration to linear convergence once the factor rank matches the true rank, validated numerically and shown to prevent overfitting in overparameterized settings.

ABSTRACT

We study the robust recovery of a low-rank matrix from sparsely and grossly corrupted Gaussian measurements, with no prior knowledge on the intrinsic rank. We consider the robust matrix factorization approach. We employ a robust $\\ell_1$ loss function and deal with the challenge of the unknown rank by using an overspecified factored representation of the matrix variable. We then solve the associated nonconvex nonsmooth problem using a subgradient method with diminishing stepsizes. We show that under a regularity condition on the sensing matrices and corruption, which we call restricted direction preserving property (RDPP), even with rank overspecified, the subgradient method converges to the exact low-rank solution at a sublinear rate. Moreover, our result is more general in the sense that it automatically speeds up to a linear rate once the factor rank matches the unknown rank. On the other hand, we show that the RDPP condition holds under generic settings, such as Gaussian measurements under independent or adversarial sparse corruptions, where the result could be of independent interest. Both the exact recovery and the convergence rate of the proposed subgradient method are numerically verified in the overspecified regime. Moreover, our experiment further shows that our particular design of diminishing stepsize effectively prevents overfitting for robust recovery under overparameterized models, such as robust matrix sensing and learning robust deep image prior. This regularization effect is worth further investigation.

Motivation & Objective

• Address the challenge of robust low-rank matrix recovery when the true rank is unknown.
• Develop a nonconvex, nonsmooth optimization method that handles sparse, gross corruptions via an $ε$-subgradient method with diminishing stepsizes.
• Establish exact recovery guarantees under rank overspecification, where the factorization dimension $k$ exceeds the true rank $r$.
• Provide theoretical convergence rates that improve from sublinear to linear once the factor rank matches the true rank.
• Demonstrate the regularization effect of diminishing stepsizes in preventing overfitting in overparameterized models like robust matrix sensing and deep image prior.

Proposed method

• Formulate the robust recovery problem using a factored, nonconvex, nonsmooth objective with an $ε$-subgradient method.
• Employ a robust $´\ell_1$ loss function to handle sparse corruptions in measurements.
• Use an overspecified factorization $X = FF^T$ with $F \in \mathbb{R}^{d \times k}$, $k > r$, to avoid prior knowledge of the true rank $r$.
• Apply a subgradient method with diminishing stepsizes to solve the nonconvex, nonsmooth optimization problem.
• Introduce spectral initialization to improve convergence and stability.
• Establish convergence under the restricted direction preserving property (RDPP), which holds generically for Gaussian measurements and sparse corruptions.

Experimental results

Research questions

• RQ1 Can exact recovery be achieved in the rank-overspecified regime where $k > r$ using a nonconvex, nonsmooth optimization method?
• RQ2 Does the subgradient method with diminishing stepsizes converge to the true low-rank matrix under the RDPP condition?
• RQ3 What is the convergence rate of the subgradient method, and does it accelerate when the factor rank matches the true rank?
• RQ4 How does the diminishing stepsize schedule prevent overfitting in overparameterized models such as robust matrix sensing and deep image prior?
• RQ5 Under what conditions does the RDPP condition hold, and is it satisfied in practical settings like Gaussian measurements with sparse corruptions?

Key findings

• Exact recovery of the true low-rank matrix $X_{\natural}$ is achieved with high probability when $m = \tilde{\mathcal{O}}(dk^3)$ measurements, even under rank overspecification.
• The subgradient method converges to the true solution at a sublinear rate $\mathcal{O}(1/t)$ when $k > r$, under the RDPP condition.
• The convergence rate accelerates to linear once the factor rank $k$ matches the true rank $r$, due to improved curvature properties.
• The RDPP condition holds generically for Gaussian measurement matrices and sparse corruptions, including both independent and adversarial corruption models.
• The diminishing stepsize schedule effectively prevents overfitting in overparameterized models, such as robust matrix sensing and deep image prior, demonstrating a regularization effect.