[Paper Review] Generalized power method for sparse principal component analysis

Michel Journée, Yurii Nesterov+2 more|arXiv (Cornell University)|Nov 28, 2008

Image and Signal Denoising MethodsComputer Science31 references500 citations

TL;DR

This paper proposes a generalized power method for sparse principal component analysis (sparse PCA) that reformulates nonconvex sparse PCA problems into convex optimization tasks, enabling efficient computation via gradient ascent. The method achieves superior performance in solution quality and convergence speed compared to existing algorithms, especially on high-dimensional and biological data sets, with block formulations delivering richer biological interpretations.

ABSTRACT

In this paper we develop a new approach to sparse principal component analysis (sparse PCA). We propose two single-unit and two block optimization formulations of the sparse PCA problem, aimed at extracting a single sparse dominant principal component of a data matrix, or more components at once, respectively. While the initial formulations involve nonconvex functions, and are therefore computationally intractable, we rewrite them into the form of an optimization program involving maximization of a convex function on a compact set. The dimension of the search space is decreased enormously if the data matrix has many more columns (variables) than rows. We then propose and analyze a simple gradient method suited for the task. It appears that our algorithm has best convergence properties in the case when either the objective function or the feasible set are strongly convex, which is the case with our single-unit formulations and can be enforced in the block case. Finally, we demonstrate numerically on a set of random and gene expression test problems that our approach outperforms existing algorithms both in quality of the obtained solution and in computational speed.

Motivation & Objective

• To develop a more efficient and interpretable approach to sparse PCA that balances variance explanation and sparsity in loading vectors.
• To address the computational intractability of nonconvex sparse PCA formulations by transforming them into convex optimization problems.
• To improve convergence and performance over existing algorithms through single-unit and block optimization formulations.
• To enable simultaneous extraction of multiple sparse components using block methods, enhancing interpretability in high-dimensional data.
• To demonstrate superior performance on random and real-world biological data, particularly in capturing biologically meaningful patterns.

Proposed method

• Proposes two single-unit and two block optimization formulations of sparse PCA, transforming nonconvex problems into maximization of convex functions over compact sets.
• Reformulates sparse PCA as maximizing a convex function subject to sparsity and unit-norm constraints, leveraging strong convexity for convergence guarantees.
• Applies a simple gradient ascent algorithm tailored for the reformulated problem, with convergence analysis under strong convexity assumptions.
• Employs a dimensionality reduction strategy when the data matrix has more columns than rows, significantly reducing the search space.
• Uses a certificate of optimality in block formulations to ensure solution quality, especially when eigenvalues are close.
• Implements a greedy heuristic with optimality certification for improved performance on ill-posed problems.

Experimental results

Research questions

• RQ1 Can sparse PCA be reformulated as a convex optimization problem to improve computational tractability and convergence?
• RQ2 How does the performance of the proposed generalized power method compare to existing sparse PCA algorithms in terms of solution quality and speed?
• RQ3 Does the block formulation of sparse PCA yield better interpretability and performance on high-dimensional biological data?
• RQ4 What role does strong convexity play in ensuring fast convergence of the proposed gradient-based algorithm?
• RQ5 Can the proposed method extract multiple sparse components simultaneously with improved accuracy compared to sequential deflation?

Key findings

• The proposed generalized power method outperforms existing algorithms in both computational speed and solution quality on random and gene expression test problems.
• The block formulations deliver the richest biological interpretation of components, particularly in gene expression data, due to better capture of meaningful biological patterns.
• The algorithm exhibits superior convergence properties when either the objective function or the feasible set is strongly convex, which is ensured in the single-unit formulations.
• The method achieves significant dimensionality reduction in the search space when the data matrix has more variables than samples, improving scalability.
• Numerical experiments confirm that the proposed approach systematically outperforms state-of-the-art methods in trade-off between explained variance and sparsity.
• Theoretical analysis confirms that the gradient method converges efficiently under strong convexity, with explicit bounds on the convergence rate.