[Paper Review] Sparse Generalized Eigenvalue Problem: Optimal Statistical Rates via Truncated Rayleigh Flow

Kean Ming Tan, Zhaoran Wang+2 more|arXiv (Cornell University)|Apr 29, 2016

Sparse and Compressive Sensing Techniques50 references3 citations

TL;DR

This paper proposes a two-stage computational framework for solving the sparse generalized eigenvalue problem (GEP) in high-dimensional settings, where the leading generalized eigenvector is assumed to be sparse. It introduces the Truncated Rayleigh Flow (Rifle) algorithm, which combines convex relaxation for initialization with nonconvex gradient ascent and hard thresholding, achieving linear convergence to a solution with optimal statistical rates without requiring structural assumptions on the input matrices A and B.

ABSTRACT

Sparse generalized eigenvalue problem (GEP) plays a pivotal role in a large family of high-dimensional statistical models, including sparse Fisher's discriminant analysis, canonical correlation analysis, and sufficient dimension reduction. Sparse GEP involves solving a non-convex optimization problem. Most existing methods and theory in the context of specific statistical models that are special cases of the sparse GEP require restrictive structural assumptions on the input matrices. In this paper, we propose a two-stage computational framework to solve the sparse GEP. At the first stage, we solve a convex relaxation of the sparse GEP. Taking the solution as an initial value, we then exploit a nonconvex optimization perspective and propose the truncated Rayleigh flow method (Rifle) to estimate the leading generalized eigenvector. We show that Rifle converges linearly to a solution with the optimal statistical rate of convergence for many statistical models. Theoretically, our method significantly improves upon the existing literature by eliminating structural assumptions on the input matrices for both stages. To achieve this, our analysis involves two key ingredients: (i) a new analysis of the gradient based method on nonconvex objective functions, and (ii) a fine-grained characterization of the evolution of sparsity patterns along the solution path. Thorough numerical studies are provided to validate the theoretical results.

Motivation & Objective

• To address the challenge of estimating a sparse leading generalized eigenvector in high-dimensional settings where d > n and the normalization matrix B̂ is singular.
• To develop a computationally efficient and statistically optimal algorithm for sparse GEP that avoids restrictive assumptions on the population matrices A and B.
• To establish theoretical guarantees for convergence and estimation error in the absence of low-rank or positive-semidefinite assumptions on A and B.
• To generalize existing methods for specific models like sparse CCA, FDA, and SDR into a unified framework with broader applicability.
• To validate the method through thorough numerical studies under various high-dimensional scenarios.

Proposed method

• The method uses a two-stage framework: first solving a convex relaxation of the sparse GEP to obtain an initial estimate v₀.
• The second stage applies the Truncated Rayleigh Flow (Rifle) algorithm, which alternates between gradient ascent on the generalized Rayleigh quotient and hard thresholding to maintain sparsity.
• At each iteration, the algorithm performs a gradient step on vᵀÂv / vᵀB̂v, followed by truncation to keep only the top k entries by magnitude.
• The tuning parameter k is selected such that k = Cs for some C > 1, ensuring the solution path maintains correct sparsity support.
• Theoretical analysis relies on a novel analysis of gradient methods on nonconvex objectives and a fine-grained characterization of sparsity pattern evolution along the solution path.
• The method handles ill-conditioned B̂ matrices arising from finite-sample estimation by avoiding matrix inversion and instead using truncated updates.

Experimental results

Research questions

• RQ1 Can a two-stage algorithm achieve optimal statistical rates for the sparse generalized eigenvalue problem without requiring structural assumptions on the input matrices A and B?
• RQ2 Does the Truncated Rayleigh Flow (Rifle) algorithm converge linearly to a solution with optimal estimation error in high-dimensional settings?
• RQ3 How does the evolution of sparsity patterns along the solution path affect the convergence and statistical performance of nonconvex optimization in sparse GEP?
• RQ4 Can the proposed method be applied to a broad class of statistical models, including sparse CCA, FDA, and SDR, with unified theoretical guarantees?
• RQ5 What is the impact of tuning parameter selection on sparsity and estimation accuracy, and can thresholding strategies replace fixed k in practice?

Key findings

• The Rifle algorithm achieves linear convergence to a solution with optimal statistical rate, as shown by the error bound ‖vᵗ − v⁎‖₂ ≤ νᵗ‖v⁰ − v⁎‖₂ + O(√(ρ(E_A,2k+s)² + ρ(E_B,2k+s)²)/ξ(A,B)).
• Theoretical guarantees are established without requiring A to be low-rank, positive semidefinite, or known in rank, significantly generalizing prior work.
• The method achieves optimal statistical error rate even when the normalization matrix B̂ is singular due to high dimensionality and finite sample effects.
• Numerical studies confirm that the soft-thresholding variant (Soft-Rifle) performs comparably to the hard-thresholding Rifle method, suggesting robustness to tuning strategy.
• The estimation error for Rifle with k=40 is 0.048 (averaged over 50 runs), with estimated sparsity matching the true s=40, demonstrating high accuracy.
• The method is scalable in theory, with total computational complexity O(d³) + O(kd + d), though the convex relaxation stage remains a computational bottleneck.