[Paper Review] Eigenvectors from Eigenvalues Sparse Principal Component Analysis (EESPCA)

H. Robert Frost|PubMed|Jun 2, 2020

Sparse and Compressive Sensing Techniques36 references4 citations

TL;DR

This paper introduces EESPCA, a novel sparse principal component analysis method that leverages the eigenvector-eigenvalue identity to compute sparse loadings efficiently. It achieves state-of-the-art sparsity estimation accuracy and up to 10x faster computation than existing methods like SPC, TPower, and rifle, while maintaining low out-of-sample reconstruction and PC estimation error.

ABSTRACT

We present a novel technique for sparse principal component analysis. This method, named Eigenvectors from Eigenvalues Sparse Principal Component Analysis (EESPCA), is based on the formula for computing squared eigenvector loadings of a Hermitian matrix from the eigenvalues of the full matrix and associated sub-matrices. We explore two versions of the EESPCA method: a version that uses a fixed threshold for inducing sparsity and a version that selects the threshold via cross-validation. Relative to the state-of-the-art sparse PCA methods of Witten et al., Yuan & Zhang and Tan et al., the fixed threshold EESPCA technique offers an order-of-magnitude improvement in computational speed, does not require estimation of tuning parameters via cross-validation, and can more accurately identify true zero principal component loadings across a range of data matrix sizes and covariance structures. Importantly, the EESPCA method achieves these benefits while maintaining out-of-sample reconstruction error and PC estimation error close to the lowest error generated by all evaluated approaches. EESPCA is a practical and effective technique for sparse PCA with particular relevance to computationally demanding statistical problems such as the analysis of high-dimensional data sets or application of statistical techniques like resampling that involve the repeated calculation of sparse PCs.

Motivation & Objective

• To develop a computationally efficient sparse PCA method that accurately identifies zero loadings in principal components.
• To address the high computational cost of existing sparse PCA techniques, especially in high-dimensional or resampling-intensive settings.
• To improve sparsity estimation accuracy without relying on time-consuming cross-validation for tuning parameters.
• To maintain low out-of-sample reconstruction and PC estimation error comparable to the best-performing methods like TPower.
• To provide a practical alternative for large-scale genomic and high-dimensional data analysis where speed and sparsity are critical.

Proposed method

• EESPCA uses the eigenvector-eigenvalue identity to compute squared eigenvector loadings from eigenvalues of the full matrix and its sub-matrices.
• A fixed threshold of $1/\sqrt{p}$ is applied to induce sparsity in the estimated principal component loadings.
• An alternative version, EESPCA.cv, selects the sparsity threshold via cross-validation to minimize out-of-sample reconstruction error.
• The method avoids iterative optimization or tuning parameter estimation, relying instead on analytical eigenvalue-based loadings.
• It computes sparse loadings directly from the eigenstructure of the sample covariance matrix, enabling rapid computation.
• The approach is grounded in the recently rediscovered identity by Denton et al. (2021), enabling exact computation of squared loadings from eigenvalues alone.

Experimental results

Research questions

• RQ1 Can the eigenvector-eigenvalue identity be effectively leveraged to create a faster and more accurate sparse PCA method?
• RQ2 How does EESPCA compare to state-of-the-art methods like SPC, TPower, and rifle in terms of sparsity estimation accuracy across diverse data structures?
• RQ3 What is the trade-off between computational speed and estimation accuracy when using a fixed threshold versus cross-validation for sparsity in EESPCA?
• RQ4 How does EESPCA perform in terms of out-of-sample reconstruction error and PC estimation error relative to existing methods?
• RQ5 In what scenarios does EESPCA outperform other sparse PCA methods, particularly in high-dimensional or resampling-intensive applications?

Key findings

• EESPCA achieves an order-of-magnitude improvement in computational speed compared to SPC, TPower, and rifle, making it highly suitable for large-scale or repeated analyses.
• The fixed-threshold EESPCA method more accurately identifies true zero loadings across various data sizes and covariance structures than SPC, TPower, and rifle.
• EESPCA maintains out-of-sample reconstruction error and PC estimation error close to the lowest values achieved by TPower, the best-performing method in this regard.
• For cases with limited sparsity (i.e., fewer than 50% zero loadings), EESPCA’s performance deteriorates significantly, but EESPCA.cv performs comparably to TPower.
• When computational speed is critical and significant sparsity is expected, EESPCA is the preferred method over EESPCA.cv, SPC, SPC.1se, TPower, and rifle.
• In scenarios where minimal estimation error is the priority and speed is not a constraint, TPower remains the best-performing method overall.