[Paper Review] Augmented sparse principal component analysis for high dimensional data

Debashis Paul, Iain M. Johnstone|arXiv (Cornell University)|Feb 6, 2012

Sparse and Compressive Sensing TechniquesEngineering36 references70 citations

TL;DR

This paper proposes an augmented sparse principal component analysis (SPCA) method for estimating leading eigenvectors of high-dimensional covariance matrices under $l^q$-sparsity constraints. By combining coordinate selection with PCA, the method achieves the optimal minimax convergence rate under a spiked covariance model, while also establishing theoretical lower bounds and conditions under which standard PCA is minimax optimal.

ABSTRACT

We study the problem of estimating the leading eigenvectors of a high-dimensional population covariance matrix based on independent Gaussian observations. We establish lower bounds on the rates of convergence of the estimators of the leading eigenvectors under $l^q$-sparsity constraints when an $l^2$ loss function is used. We also propose an estimator of the leading eigenvectors based on a coordinate selection scheme combined with PCA and show that the proposed estimator achieves the optimal rate of convergence under a sparsity regime. Moreover, we establish that under certain scenarios, the usual PCA achieves the minimax convergence rate.

Motivation & Objective

• Address the challenge of estimating leading eigenvectors in high-dimensional settings where dimension $N$ grows with sample size $n$.
• Study the minimax estimation rates of eigenvectors under $l^q$-sparsity constraints for $q \in (0,2]$.
• Propose a novel estimator that combines coordinate selection with PCA to achieve optimal convergence rates.
• Establish theoretical lower bounds on the estimation error for eigenvectors under sparsity.
• Identify conditions under which standard PCA achieves the minimax rate, despite lack of sparsity enforcement.

Proposed method

• Formulate the estimation problem under the spiked covariance model, where only the first $M$ eigenvalues are distinct from the noise level $\sigma^2$.
• Introduce a coordinate selection scheme that identifies relevant variables by thresholding sample loadings or projections.
• Apply standard PCA on the selected subset of variables to estimate the leading eigenvectors.
• Use $l^q$-norm constraints to model sparsity in the true eigenvectors, with $q \in (0,2]$.
• Derive minimax lower bounds on the $l^2$-loss for eigenvector estimation using local asymptotic normality and information-theoretic arguments.
• Analyze the convergence rate of the proposed estimator and compare it to the minimax lower bound.

Experimental results

Research questions

• RQ1 What is the minimax rate of convergence for estimating leading eigenvectors under $l^q$-sparsity constraints in high-dimensional settings?
• RQ2 Can a coordinate selection-based PCA method achieve the optimal minimax rate of convergence?
• RQ3 Under what conditions does standard PCA achieve the minimax rate despite not enforcing sparsity?
• RQ4 How does the proposed augmented SPCA method compare to existing sparse PCA approaches in terms of theoretical optimality?
• RQ5 What is the impact of the sparsity level $M$ and the noise variance $\sigma^2$ on the estimation error?

Key findings

• The paper establishes a lower bound on the $l^2$-loss for estimating leading eigenvectors under $l^q$-sparsity, showing that the minimax rate depends on the sparsity level $M$ and the dimension $N$.
• The proposed augmented SPCA estimator achieves the optimal minimax rate of convergence under the $l^2$-loss when $q \in (0,2]$.
• Under certain conditions on the eigenvalue gap $\ell_M - \sigma^2$, standard PCA achieves the minimax rate, even without explicit sparsity enforcement.
• The coordinate selection step effectively identifies the true support of the leading eigenvectors with high probability under appropriate regularity conditions.
• The minimax rate is of order $\sqrt{M \log N / n}$ for $q=1$, and improves as $q$ increases toward 2, reflecting the benefit of sparsity.
• The theoretical results are validated through a rigorous analysis of the asymptotic behavior of the estimator in the high-dimensional regime $N \to \infty$, $n \to \infty$, with $N/n \to c \in (0, \infty)$.