[Paper Review] Bridging Convex and Nonconvex Optimization in Robust PCA: Noise, Outliers, and Missing Data

Yuxin Chen, Jianqing Fan+2 more|arXiv (Cornell University)|Jan 15, 2020

Sparse and Compressive Sensing Techniques78 references4 citations

TL;DR

This paper establishes near-optimal statistical guarantees for convex robust PCA under random noise, gross outliers, and missing data by bridging convex relaxation with an auxiliary nonconvex optimization framework. It proves that, under mild conditions, the convex program achieves near-minimax optimal error rates in both Frobenius and ℓ∞ norms, even when nearly a constant fraction of entries are arbitrarily corrupted.

ABSTRACT

This paper delivers improved theoretical guarantees for the convex programming approach in low-rank matrix estimation, in the presence of (1) random noise, (2) gross sparse outliers, and (3) missing data. This problem, often dubbed as robust principal component analysis (robust PCA), finds applications in various domains. Despite the wide applicability of convex relaxation, the available statistical support (particularly the stability analysis vis-à-vis random noise) remains highly suboptimal, which we strengthen in this paper. When the unknown matrix is well-conditioned, incoherent, and of constant rank, we demonstrate that a principled convex program achieves near-optimal statistical accuracy, in terms of both the Euclidean loss and the $\ell_{\infty}$ loss. All of this happens even when nearly a constant fraction of observations are corrupted by outliers with arbitrary magnitudes. The key analysis idea lies in bridging the convex program in use and an auxiliary nonconvex optimization algorithm, and hence the title of this paper.

Motivation & Objective

• Address the theoretical gap in convex robust PCA under random noise, where existing stability analyses are suboptimal.
• Provide improved statistical guarantees for low-rank matrix estimation when data are corrupted by random noise, gross sparse outliers, and missing entries.
• Demonstrate that convex relaxation achieves near-optimal recovery accuracy even when a constant fraction of observations are corrupted by outliers of arbitrary magnitude.
• Bridge the analysis of convex relaxation with an auxiliary nonconvex optimization procedure to derive tighter error bounds.
• Establish both Frobenius and ℓ∞ error guarantees under well-conditioned, incoherent, and constant-rank assumptions on the underlying low-rank matrix.

Proposed method

• Introduce a principled convex program minimizing nuclear norm plus ℓ1-norm of the sparse component, subject to data consistency on observed entries.
• Use a leave-one-out analysis framework to study the convergence and stability of the nonconvex optimization procedure used as a proxy for the convex solution.
• Establish equivalence between the convex and nonconvex solutions by analyzing the difference between iterates via perturbation bounds.
• Apply probabilistic and random matrix theory tools to control the behavior of the estimation error under sub-Gaussian noise and sparse corruption.
• Derive crude error bounds for the convex relaxation and refine them through nonconvex algorithm analysis, leading to tighter statistical guarantees.
• Leverage the structure of the optimization problem to show that the difference between the true and estimated components is small in both ℓ∞ and Frobenius norms.

Experimental results

Research questions

• RQ1 Can the convex relaxation approach in robust PCA achieve near-optimal statistical accuracy under random noise, gross outliers, and missing data?
• RQ2 What is the tightest possible error bound achievable by convex relaxation in the presence of arbitrary-magnitude sparse outliers and sub-Gaussian noise?
• RQ3 How can the gap between theoretical guarantees and practical performance in convex robust PCA be closed using nonconvex analysis?
• RQ4 To what extent can the convex program maintain stability and accuracy when nearly a constant fraction of entries are corrupted?
• RQ5 Under what conditions does the convex solution closely approximate the solution of a nonconvex optimization procedure?

Key findings

• The convex program achieves near-optimal Frobenius error rate of order $\sqrt{r(n_1+n_2)\log n}/n$ under sub-Gaussian noise, matching the minimax lower bound up to logarithmic factors.
• The ℓ∞ error bound is also near-optimal, scaling as $\sqrt{\log n}/n$ for each entry, even when a constant fraction of entries are grossly corrupted.
• The method remains stable and accurate when the fraction of corrupted entries approaches a constant, provided the outlier matrix is sparse and the low-rank matrix is incoherent.
• The analysis establishes that the convex relaxation solution is within a small neighborhood of the true low-rank matrix, with the error controlled via leave-one-out techniques.
• The equivalence between the convex and nonconvex solutions is proven under mild assumptions, showing that the nonconvex algorithm can be used to analyze the convex one.
• The results hold under the condition that $n^2 p \rho_{\text{aug}} \gg \mu r n \log n$, ensuring sufficient sampling for stable recovery.