[Paper Review] Robust PCA by Manifold Optimization

Teng Zhang, Yi Yang|arXiv (Cornell University)|Aug 1, 2017

Sparse and Compressive Sensing Techniques36 references19 citations

TL;DR

This paper proposes two manifold optimization algorithms for robust PCA that exploit the low-rank structure of the data matrix by parameterizing it via the Burer-Monterio decomposition. With proper initialization, the algorithms converge linearly to the true low-rank matrix and reduce theoretical dependence on the condition number of the underlying matrix compared to prior methods.

ABSTRACT

Robust PCA is a widely used statistical procedure to recover a underlying low-rank matrix with grossly corrupted observations. This work considers the problem of robust PCA as a nonconvex optimization problem on the manifold of low-rank matrices, and proposes two algorithms (for two versions of retractions) based on manifold optimization. It is shown that, with a proper designed initialization, the proposed algorithms are guaranteed to converge to the underlying low-rank matrix linearly. Compared with a previous work based on the Burer-Monterio decomposition of low-rank matrices, the proposed algorithms reduce the dependence on the conditional number of the underlying low-rank matrix theoretically. Simulations and real data examples confirm the competitive performance of our method.

Motivation & Objective

• Address the challenge of recovering a low-rank matrix from grossly corrupted observations in high-dimensional data.
• Overcome the limitations of convex relaxation methods, which suffer from high computational complexity ($O(n_1n_2\min(n_1,n_2))$ per iteration).
• Develop a nonconvex optimization framework on the manifold of low-rank matrices to achieve faster and more scalable recovery.
• Reduce theoretical dependence on the condition number of the underlying low-rank matrix compared to prior nonconvex methods like Burer-Monterio-based approaches.
• Ensure linear convergence with a proper initialization strategy, improving robustness and efficiency in practical settings.

Proposed method

• Formulate robust PCA as a nonconvex optimization problem on the manifold of low-rank matrices, parameterizing the low-rank matrix $\boldsymbol{L}^*$ as $\boldsymbol{U}\boldsymbol{V}^T$.
• Use manifold optimization techniques, including retractions and vector transport, to navigate the nonconvex manifold structure efficiently.
• Design two algorithms based on different retraction schemes to ensure numerical stability and convergence guarantees.
• Apply gradient descent on the manifold of low-rank matrices, leveraging the intrinsic geometry to avoid parameterization redundancy.
• Introduce a tailored initialization procedure that ensures the iterates remain within a neighborhood of the true low-rank matrix.
• Use concentration inequalities and net arguments to bound the impact of sparse noise $\boldsymbol{S}^*$ on the optimization path, ensuring robustness.

Experimental results

Research questions

• RQ1 Can robust PCA be solved efficiently via nonconvex optimization on the manifold of low-rank matrices with theoretical convergence guarantees?
• RQ2 How does the proposed method reduce dependence on the condition number of the underlying low-rank matrix compared to existing Burer-Monterio-based approaches?
• RQ3 What initialization strategy ensures linear convergence of the optimization algorithm to the true low-rank matrix?
• RQ4 Does the manifold optimization framework maintain robustness under high levels of sparse corruption in the data?
• RQ5 Can the algorithm achieve faster convergence and lower computational cost than convex relaxation or standard nonconvex methods?

Key findings

• The proposed algorithms achieve linear convergence to the true low-rank matrix $\boldsymbol{L}^*$ when initialized properly, with convergence rate independent of the condition number $\kappa$.
• Theoretical analysis shows that the method reduces dependence on $\kappa$ compared to prior Burer-Monterio-based methods, which scale as $O(1/\kappa^2)$.
• The algorithm achieves a per-iteration complexity of $O(r n_1 n_2)$, matching the best nonconvex methods, but with improved theoretical robustness.
• Simulations and real data experiments confirm competitive performance in terms of recovery accuracy and computational efficiency.
• With high probability, the algorithm recovers $\boldsymbol{L}^*$ when the fraction of corrupted entries $\gamma = O(1/\mu^2 r)$, matching the best-known theoretical bounds.
• Theoretical bounds on the noise term $\boldsymbol{N}^*$ are derived using net covering and concentration inequalities, ensuring that the noise does not disrupt convergence.