[Paper Review] Approximation Algorithms for Sparse Principal Component Analysis

Agniva Chowdhury, Petros Drineas+2 more|arXiv (Cornell University)|Jun 23, 2020

Sparse and Compressive Sensing Techniques36 references4 citations

TL;DR

This paper presents two thresholding-based approximation algorithms for Sparse Principal Component Analysis (SPCA) that achieve provably accurate, polynomial-time solutions without restrictive assumptions on the input covariance matrix. The first uses SVD with thresholding for fast, practical performance; the second combines semidefinite programming with a novel two-step deterministic thresholding scheme, offering the strongest theoretical guarantees that closely match empirical results, effectively bridging the theory-practice gap.

ABSTRACT

Principal component analysis (PCA) is a widely used dimension reduction technique in machine learning and multivariate statistics. To improve the interpretability of PCA, various approaches to obtain sparse principal direction loadings have been proposed, which are termed Sparse Principal Component Analysis (SPCA). In this paper, we present thresholding as a provably accurate, polynomial time, approximation algorithm for the SPCA problem, without imposing any restrictive assumptions on the input covariance matrix. Our first thresholding algorithm using the Singular Value Decomposition is conceptually simple; is faster than current state-of-the-art; and performs well in practice. On the negative side, our (novel) theoretical bounds do not accurately predict the strong practical performance of this approach. The second algorithm solves a well-known semidefinite programming relaxation and then uses a novel, two step, deterministic thresholding scheme to compute a sparse principal vector. It works very well in practice and, remarkably, this solid practical performance is accurately predicted by our theoretical bounds, which bridge the theory-practice gap better than current state-of-the-art.

Motivation & Objective

• To develop efficient, provably accurate approximation algorithms for Sparse PCA (SPCA) without restrictive assumptions on the input covariance matrix.
• To address the theory-practice gap in SPCA by designing algorithms whose theoretical bounds accurately predict real-world performance.
• To provide runtime-vs-accuracy trade-offs through a simple, fast SVD-based thresholding approach.
• To improve theoretical guarantees for SPCA by introducing a novel two-step deterministic thresholding scheme on SDP relaxations.
• To extend the applicability of the methods to sparse kernel PCA.

Proposed method

• The first algorithm, spca-svd, applies thresholding to the top singular vectors obtained via SVD of the input covariance matrix A, producing a sparse vector with provable approximation guarantees.
• The second algorithm, spca-sdp, solves a well-known semidefinite programming (SDP) relaxation of the SPCA problem and then applies a two-step deterministic thresholding process to extract a sparse principal component.
• Theoretical analysis bounds the approximation error in terms of constants α and β derived from the spectrum of the input matrix A, which are empirically close to 1 in real datasets.
• The algorithms are designed to maintain sparsity (‖x‖₀ ≤ k) while maximizing the Rayleigh quotient xᵀAx under unit norm constraint.
• Theoretical bounds are derived using properties of the singular value decomposition and the structure of the SDP relaxation, ensuring polynomial-time complexity.
• The approach is extended to sparse kernel PCA by applying the same thresholding framework to kernel matrices.

Experimental results

Research questions

• RQ1 Can thresholding be used as a provably accurate, polynomial-time approximation algorithm for SPCA without restrictive assumptions on the input covariance matrix?
• RQ2 Why does the SVD-based thresholding approach (spca-svd) perform strongly in practice despite weak theoretical bounds predicting poor performance?
• RQ3 Can a novel two-step deterministic thresholding scheme on SDP relaxations provide both strong theoretical guarantees and accurate prediction of practical performance?
• RQ4 How do the theoretical bounds of spca-svd, spca-lowrank, and spca-sdp compare in practice, especially regarding the theory-practice gap?
• RQ5 Can the proposed algorithms approximately recover the support of the optimal sparse principal component vector x* rather than just the optimal value Z*?

Key findings

• The spca-svd algorithm achieves fast runtime and strong practical performance but has theoretical bounds that fail to predict its accuracy, highlighting a significant theory-practice gap.
• The spca-sdp algorithm, based on SDP relaxation with two-step thresholding, provides theoretical bounds that are consistently close to 1, accurately predicting its high empirical accuracy.
• For real datasets like HGDP/HapMap and gene expression data, the constants α and β in the theoretical bound are very close to 1, explaining the strong performance of spca-sdp.
• The theoretical bounds of spca-svd remain meaningless (negative) for small sparsity levels, while spca-sdp’s bounds remain positive and tight across all tested sparsity levels.
• The spca-sdp algorithm outperforms both spca-svd and spca-lowrank in theoretical accuracy guarantees, especially in scenarios where the spectrum of A is well-behaved.
• Empirical results show that spca-sdp’s theoretical bounds are nearly indistinguishable from the actual accuracy ratio in practice, effectively closing the theory-practice gap.