[Paper Review] Sum-of-squares lower bounds for Sparse PCA

Tengyu Ma, Avi Wigderson|arXiv (Cornell University)|Dec 7, 2015

Sparse and Compressive Sensing Techniques44 references30 citations

TL;DR

This paper establishes that degree-4 Sum-of-Squares (SoS) relaxations cannot close the statistical-computational gap in Sparse PCA, proving that even this higher-degree convex relaxation requires n ≈ k² samples to detect a planted k-sparse unit vector—matching the sample complexity of lower-degree methods. The result highlights inherent limitations of SoS methods in high-dimensional sparse estimation.

ABSTRACT

This paper establishes a statistical versus computational trade-off for solving a basic high-dimensional machine learning problem via a basic convex relaxation method. Specifically, we consider the Sparse Principal Component Analysis (Sparse PCA) problem, and the family of Sum-of-Squares (SoS, aka Lasserre/Parillo) convex relaxations. It was well known that in large dimension p, a planted k-sparse unit vector can be in principle detected using only n ≈ k log p (Gaussian or Bernoulli) samples, but all efficient (polynomial time) algorithms known require n ≈ k2 samples. It was also known that this quadratic gap cannot be improved by the the most basic semi-definite (SDP, aka spectral) relaxation, equivalent to a degree-2 SoS algorithms. Here we prove that also degree-4 SoS algorithms cannot improve this quadratic gap. This average-case lower bound adds to the small collection of hardness results in machine learning for this powerful family of convex relaxation algorithms. Moreover, our design of moments (or pseudo-expectations) for this lower bound is quite different than previous lower bounds. Establishing lower bounds for higher degree SoS algorithms for remains a challenging problem.

Motivation & Objective

• To investigate whether higher-degree Sum-of-Squares (SoS) relaxations can close the statistical-computational gap in Sparse PCA.
• To determine if degree-4 SoS algorithms improve upon the n ≈ k² sample complexity required by lower-degree methods.
• To establish an average-case lower bound for SoS algorithms in the context of sparse high-dimensional estimation.
• To develop a novel construction of moments (pseudo-expectations) that enables this lower bound, differing from prior approaches.
• To contribute to the growing body of evidence that convex relaxations like SoS face intrinsic limitations in computational efficiency for certain high-dimensional problems.

Proposed method

• The authors analyze the performance of degree-4 Sum-of-Squares (SoS) relaxations on the Sparse PCA problem.
• They construct a specific distribution of planted sparse vectors and define a pseudo-expectation operator that satisfies the SoS constraints up to degree 4.
• The construction of moments (pseudo-expectations) is designed to mimic the behavior of a true distribution while maintaining consistency with low-degree moments.
• The proof relies on showing that no degree-4 SoS solution can distinguish the planted sparse vector from noise when n ≈ k log p samples are available.
• The approach diverges from prior lower bounds by using a non-standard moment design tailored to the structure of the Sparse PCA problem.
• The argument combines tools from pseudocalibration and moment matrix analysis to derive a contradiction when assuming successful detection with sub-quadratic samples.

Experimental results

Research questions

• RQ1 Can degree-4 Sum-of-Squares relaxations detect a k-sparse unit vector in high dimensions with n ≈ k log p samples?
• RQ2 Is there a fundamental computational barrier that prevents SoS methods from achieving the information-theoretic sample complexity in Sparse PCA?
• RQ3 How does the design of pseudo-expectations for SoS lower bounds differ when applied to Sparse PCA compared to previous problems?
• RQ4 Can higher-degree SoS relaxations overcome the k² sample complexity gap observed in spectral and degree-2 SoS methods?
• RQ5 What are the limitations of convex relaxation techniques like SoS in solving high-dimensional sparse estimation problems?

Key findings

• Degree-4 Sum-of-Squares relaxations cannot detect a planted k-sparse unit vector in high dimensions when the sample size is n ≈ k log p.
• The paper proves that even degree-4 SoS methods require n ≈ k² samples, matching the sample complexity of lower-degree spectral and SDP relaxations.
• This establishes a statistical versus computational trade-off that persists even for higher-degree SoS relaxations.
• The lower bound is achieved through a novel construction of pseudo-expectations that differs significantly from previous approaches in the literature.
• The result contributes to the understanding of inherent limitations in convex relaxation methods for high-dimensional sparse estimation.
• The findings suggest that overcoming the k² sample complexity gap may require non-convex or non-SOS-based methods.