[Paper Review] Truncated Power Method for Sparse Eigenvalue Problems

Xiao–Tong Yuan, Tong Zhang|arXiv (Cornell University)|Dec 12, 2011

Sparse and Compressive Sensing TechniquesEngineering35 references254 citations

TL;DR

This paper proposes the truncated power method, a computationally efficient algorithm for solving the sparse eigenvalue problem by combining power iteration with iterative hard thresholding to enforce sparsity. The method provably recovers sparse eigenvectors under mild conditions, offering strong theoretical guarantees and competitive empirical performance in sparse PCA and densest k-subgraph problems.

ABSTRACT

This paper considers the sparse eigenvalue problem, which is to extract dominant (largest) sparse eigenvectors with at most $k$ non-zero components. We propose a simple yet effective solution called truncated power method that can approximately solve the underlying nonconvex optimization problem. A strong sparse recovery result is proved for the truncated power method, and this theory is our key motivation for developing the new algorithm. The proposed method is tested on applications such as sparse principal component analysis and the densest $k$-subgraph problem. Extensive experiments on several synthetic and real-world large scale datasets demonstrate the competitive empirical performance of our method.

Motivation & Objective

• To develop an efficient and theoretically grounded algorithm for computing the largest k-sparse eigenvector of a symmetric positive semidefinite matrix.
• To address the non-convex, NP-hard nature of the sparse eigenvalue problem with a practical yet provably effective method.
• To provide a theoretical analysis showing that the algorithm recovers the true sparse eigenvector when the perturbation in the matrix is small relative to the sparse submatrices.
• To demonstrate the method’s effectiveness on real-world and synthetic datasets in applications like sparse PCA and densest k-subgraph detection.

Proposed method

• The method extends the classical power method by applying a hard thresholding step after each power iteration to retain only the k largest-magnitude components, enforcing sparsity.
• At each iteration, the algorithm computes $ x_{t+1} = \text{shrink}(A x_t, k) $, where $ \text{shrink}(\cdot, k) $ keeps only the k largest entries in magnitude.
• The algorithm is initialized with a random vector or a simple heuristic, and convergence is monitored via the Rayleigh quotient $ x^\top A x $.
• Theoretical analysis bounds the recovery error in terms of the spectral norm of sparse submatrices of the perturbation matrix $ E $, not the full matrix dimension $ p $.
• The method is extended to the smallest k-sparse eigenvalue problem by minimizing $ x^\top A x $ under the same constraints.
• For the densest k-subgraph problem, the method is adapted by using the adjacency matrix and applying the same truncation-based power iteration.

Experimental results

Research questions

• RQ1 Can a simple, iterative method with truncation effectively approximate the dominant k-sparse eigenvector of a symmetric matrix?
• RQ2 Under what conditions can the truncated power method provably recover the true sparse eigenvector when the matrix is perturbed?
• RQ3 How does the performance of the truncated power method compare to existing greedy and convex relaxation methods in sparse PCA and densest k-subgraph problems?
• RQ4 Does the theoretical recovery guarantee depend on the full matrix dimension or on the sparsity level and submatrix structure?

Key findings

• The truncated power method achieves competitive performance in sparse PCA, successfully recovering sparse principal components from noisy covariance matrices.
• On the densest k-subgraph problem, TPower-DkS outperforms Greedy-Feige and Greedy-Ravi in terms of subgraph density, especially on large-scale graphs like hollywood-2009.
• In the air-travel routing dataset, TPower-DkS discovered six densest 30-subgraphs with a total density of 1.14, surpassing Greedy-Feige (0.90) and Greedy-Ravi (0.99).
• The method is computationally efficient: on hollywood-2009, TPower-DkS runs in about 10 seconds per k, compared to 1 second for Greedy-Feige and significantly longer for Greedy-Ravi when k is large.
• Theoretical analysis shows that recovery error depends on the spectral norm of sparse submatrices of the perturbation $ E $, not the full dimension $ p $, which explains the strong empirical performance.
• The method is the first to provide a non-asymptotic, non-spiked-model recovery guarantee for sparse eigenvector estimation under general conditions.