[Paper Review] On the Quality of a Semidefinite Programming Bound for Sparse Principal Component Analysis

Laurent El Ghaoui|ArXiv.org|Jan 18, 2006

Sparse and Compressive Sensing Techniques4 references9 citations

TL;DR

This paper proposes a semidefinite programming (SDP) relaxation for sparse principal component analysis (SPCA) by reformulating the cardinality-penalized rank-one approximation problem as a maximum eigenvalue optimization over a box constraint. It derives a tight SDP bound using robust optimization techniques and establishes theoretical quality guarantees: the bound is exact (i.e., the relaxation gap is zero) when the diagonal entries of the covariance matrix decay rapidly, and the relative gap is bounded below by a function of the matrix structure and penalty parameter.

ABSTRACT

We examine the problem of approximating a positive, semidefinite matrix $Σ$ by a dyad $xx^T$, with a penalty on the cardinality of the vector $x$. This problem arises in sparse principal component analysis, where a decomposition of $Σ$ involving sparse factors is sought. We express this hard, combinatorial problem as a maximum eigenvalue problem, in which we seek to maximize, over a box, the largest eigenvalue of a symmetric matrix that is linear in the variables. This representation allows to use the techniques of robust optimization, to derive a bound based on semidefinite programming. The quality of the bound is investigated using a technique inspired by Nemirovski and Ben-Tal (2002).

Motivation & Objective

• To develop a tight semidefinite programming (SDP) relaxation for sparse principal component analysis (SPCA), which seeks low-rank, sparse approximations of a positive semidefinite covariance matrix.
• To analyze the quality of the SDP relaxation by deriving bounds on the relative gap between the optimal value of the original nonconvex problem and the SDP upper bound.
• To characterize conditions under which the SDP relaxation is exact (i.e., the gap is zero), particularly based on the decay rate of the diagonal entries of the covariance matrix.
• To provide a quantitative estimate of the relaxation gap as a function of the penalty parameter ρ and structural properties of the covariance matrix Σ.

Proposed method

• Reformulates the sparse PCA problem as a maximum eigenvalue problem over a box constraint: max_{x ∈ [-1,1]^n} λ_max(Σ - ρ diag(x) + diag(x)Σ), where λ_max denotes the largest eigenvalue.
• Introduces a thresholded Rayleigh quotient representation to connect the original problem to robust optimization, enabling the derivation of an SDP upper bound via duality and expectation-based bounds.
• Applies robust optimization techniques to derive an upper bound on the optimal value φ(ρ), using a probabilistic representation involving Gaussian random variables and positive part expectations.
• Derives a quality estimate θ(ρ) such that the SDP bound is at least θ(ρ) times the true optimal value, with θ(ρ) depending on the rank and structure of the covariance matrix.
• Uses the function θ_m(γ) to quantify the bound quality, where γ is a function of ρ, Σ_11, and the matrix rank m, and proves that θ_m(γ) decreases with m for fixed γ.
• Employs a lower bound on the expected positive part of a quadratic form in Gaussian variables to derive a tractable expression for the quality estimate, leveraging convexity and symmetry.

Experimental results

Research questions

• RQ1 Under what conditions is the SDP relaxation for sparse PCA exact, i.e., when does the relaxation gap vanish?
• RQ2 How does the quality of the SDP bound depend on the penalty parameter ρ in the cardinality-penalized formulation?
• RQ3 What structural properties of the covariance matrix Σ ensure a non-vanishing lower bound on the relative quality of the SDP relaxation?
• RQ4 Can a tight, analytically tractable bound be derived for the gap between the true optimal value and the SDP upper bound in sparse PCA?
• RQ5 How does the rank of the covariance matrix and the decay rate of its diagonal entries influence the tightness of the SDP relaxation?

Key findings

• The SDP relaxation is exact (i.e., the bound is tight) when the diagonal entries of Σ decay rapidly: if Σ_{22} ≤ ρ < Σ_{11}, then the relaxation gap is zero, so φ(ρ) = ψ(ρ).
• When the diagonal entries satisfy Σ_{hh} ≤ (1/(h+1))Σ_{11} for all h ≥ 2, the relaxation gap is bounded below by 1/π for all ρ < Σ_{22}, ensuring a constant relative quality guarantee.
• The quality of the SDP bound is quantified by a function θ(ρ) = θ_{m(ρ)}(γ(ρ)), where m(ρ) is the rank of the submatrix of Σ corresponding to indices with Σ_{ii} > ρ, and γ(ρ) depends on ρ, Σ_{11}, and the number of such indices.
• The bound θ(ρ) decreases with the rank m(ρ) of the active submatrix, indicating that higher-rank submatrices lead to looser bounds, but the decay is controlled by the structure of Σ.
• For random covariance matrices with fast-decaying diagonals, the quality estimate θ(ρ) remains bounded away from zero, ensuring the SDP bound is informative across a wide range of ρ.
• The paper proves that the SDP bound is at least θ(ρ) times the true optimal value, with θ(ρ) computable from the matrix structure and ρ, providing a practical and theoretical measure of relaxation quality.