[Paper Review] Robust PCA via Outlier Pursuit

Huan Xu, Constantine Caramanis+1 more|arXiv (Cornell University)|Oct 20, 2010

Sparse and Compressive Sensing Techniques26 references205 citations

TL;DR

This paper proposes Outlier Pursuit, a convex optimization method that exactly recovers the column space of a low-rank matrix and identifies corrupted columns (outliers) in a data matrix, even when outliers are arbitrary and numerous. The method uses nuclear norm minimization and $̂´1,2$-norm regularization to jointly recover the low-rank structure and detect outliers under mild incoherence and fraction-of-outliers conditions.

ABSTRACT

Singular Value Decomposition (and Principal Component Analysis) is one of the most widely used techniques for dimensionality reduction: successful and efficiently computable, it is nevertheless plagued by a well-known, well-documented sensitivity to outliers. Recent work has considered the setting where each point has a few arbitrarily corrupted components. Yet, in applications of SVD or PCA such as robust collaborative filtering or bioinformatics, malicious agents, defective genes, or simply corrupted or contaminated experiments may effectively yield entire points that are completely corrupted. We present an efficient convex optimization-based algorithm we call Outlier Pursuit, that under some mild assumptions on the uncorrupted points (satisfied, e.g., by the standard generative assumption in PCA problems) recovers the exact optimal low-dimensional subspace, and identifies the corrupted points. Such identification of corrupted points that do not conform to the low-dimensional approximation, is of paramount interest in bioinformatics and financial applications, and beyond. Our techniques involve matrix decomposition using nuclear norm minimization, however, our results, setup, and approach, necessarily differ considerably from the existing line of work in matrix completion and matrix decomposition, since we develop an approach to recover the correct column space of the uncorrupted matrix, rather than the exact matrix itself. In any problem where one seeks to recover a structure rather than the exact initial matrices, techniques developed thus far relying on certificates of optimality, will fail. We present an important extension of these methods, that allows the treatment of such problems.

Motivation & Objective

• Address the limitation of standard PCA in handling arbitrary, fully corrupted data points (outliers) that severely distort low-rank approximations.
• Develop a method that not only recovers the true low-dimensional subspace but also identifies the exact locations of corrupted columns.
• Provide theoretical guarantees for exact recovery of the column space and outlier support under natural, mild assumptions.
• Overcome the shortcomings of existing robust PCA methods, which either fail in high dimensions or cannot identify outliers.
• Extend the framework to handle noisy and partially observed data, relevant to real-world applications like collaborative filtering and bioinformatics.

Proposed method

• Formulate the problem as a matrix decomposition $ M = L_0 + C_0 $, where $ L_0 $ is low-rank and $ C_0 $ is column-sparse (outliers).
• Use a convex optimization program: minimize $ \|L\|_* + \lambda \|C\|_{1,2} $ subject to $ L + C = M $, where $ \|\cdot\|_* $ is the nuclear norm and $ \|\cdot\|_{1,2} $ is the mixed norm.
• Leverage an oracle problem to analyze recovery conditions when the target structure (column space) does not uniquely correspond to a single matrix.
• Introduce a novel analytical framework based on certificates of optimality tailored for column space recovery, not full matrix reconstruction.
• Apply the method to noisy and incomplete observation settings by modifying the constraint to $ \mathcal{P}_\Omega(L + C) = \mathcal{P}_\Omega(M) $, where $ \Omega $ is the set of observed entries.
• Use the $ \ell_2 $-norm of each column in the recovered $ C $ matrix as a heuristic to identify outliers, with thresholding applied post-optimization.

Experimental results

Research questions

• RQ1 Can we exactly recover the column space of a low-rank matrix when a fraction of its columns are arbitrarily corrupted (outliers), even without prior knowledge of the rank or outlier positions?
• RQ2 Under what conditions does the convex optimization formulation $ \min \|L\|_* + \lambda \|C\|_{1,2} $ yield exact recovery of both the column space and outlier locations?
• RQ3 How does the performance of the method scale with increasing dimensionality, especially compared to traditional robust PCA algorithms with diminishing breakdown points?
• RQ4 Can the method successfully identify outliers in the presence of noise or partial observation of the data matrix?
• RQ5 Is the method invariant to rotations of the data, and does it avoid the need for column-space incoherence assumptions common in other matrix recovery frameworks?

Key findings

• Outlier Pursuit exactly recovers the column space of $ L_0 $ and the support of $ C_0 $ under mild conditions, including bounded fraction of outliers and incoherence of the row space of $ L_0 $.
• The method achieves exact recovery even when outliers are arbitrary and not sparse in entries, unlike traditional matrix completion approaches.
• In the noisy case, Outlier Pursuit correctly identifies outliers when the signal-to-noise ratio $ \sigma/s \leq 0.3 $ for identical outliers and $ \sigma/s \leq 0.7 $ for random outliers.
• For partially observed data, the algorithm maintains high success rates even at 30% observation ratio, with performance close to the complete observation case.
• On the USPS digit dataset, the algorithm successfully identifies all 11 samples of digit '7' as outliers, with two '1' samples also flagged due to atypical writing styles.
• The method is rotation-invariant and does not require column-space incoherence, distinguishing it from prior nuclear norm-based approaches that rely on such assumptions.