[Paper Review] Statistical Limits of Convex Relaxations

Zhaoran Wang, Quanquan Gu+1 more|arXiv (Cornell University)|Mar 4, 2015

Sparse and Compressive Sensing Techniques43 references3 citations

TL;DR

This paper establishes that convex relaxations—specifically the sum-of-squares (SoS) hierarchy—sacrifice statistical optimality for computational tractability in high-dimensional sparse estimation problems. By constructing feasible solutions to SoS relaxations, the authors prove a fundamental gap of √s∗ between the information-theoretic lower bound and the achievable error rate under convex relaxations, demonstrating that computational efficiency comes at the cost of statistical performance in sparse principal submatrix and stochastic block model estimation.

ABSTRACT

Many high dimensional sparse learning problems are formulated as nonconvex optimization. A popular approach to solve these nonconvex optimization problems is through convex relaxations such as linear and semidefinite programming. In this paper, we study the statistical limits of convex relaxations. Particularly, we consider two problems: Mean estimation for sparse principal submatrix and edge probability estimation for stochastic block model. We exploit the sum-of-squares relaxation hierarchy to sharply characterize the limits of a broad class of convex relaxations. Our result shows statistical optimality needs to be compromised for achieving computational tractability using convex relaxations. Compared with existing results on computational lower bounds for statistical problems, which consider general polynomial-time algorithms and rely on computational hardness hypotheses on problems like planted clique detection, our theory focuses on a broad class of convex relaxations and does not rely on unproven hypotheses.

Motivation & Objective

• To characterize the statistical limits of convex relaxations, particularly the sum-of-squares (SoS) hierarchy, in high-dimensional sparse estimation problems.
• To investigate whether convex relaxations can achieve the information-theoretic lower bound in challenging regimes where s∗ = o(d / √log d)^(2/3) and log d = o(s∗).
• To establish a gap between the optimal statistical rate and the rate achievable via convex relaxations, without relying on unproven computational hardness assumptions.
• To demonstrate that even the tightest convex relaxations, such as the SoS hierarchy, cannot achieve optimal statistical performance when computational tractability is required.

Proposed method

• Uses the sum-of-squares (SoS) hierarchy—a sequence of increasingly tighter convex relaxations based on semidefinite programming—to approximate nonconvex sparse estimation problems.
• Constructs dual certificates based on the expansion properties of subgraphs and submatrices to verify feasibility of solutions in the SoS hierarchy.
• Applies a construction-based proof strategy rather than relying on computational reductions or hardness hypotheses.
• Defines a matrix Π(ℓ) indexed over subsets of size ℓ, with entries constructed using the number of 2ℓ×2ℓ all-one submatrices extended from a given set.
• Uses concentration inequalities and sub-Gaussian tail bounds to show that the constructed Π(ℓ) is positive semidefinite with high probability.
• Derives a lower bound on estimation error by combining the objective value of the SoS program with Markov’s inequality, showing that any estimator in the hierarchy must incur error at least C′.

Experimental results

Research questions

• RQ1 Can convex relaxations such as the SoS hierarchy achieve the information-theoretic lower bound in sparse principal submatrix and stochastic block model estimation?
• RQ2 What is the fundamental statistical cost of using convex relaxations for computational tractability in high-dimensional sparse estimation?
• RQ3 Does the performance gap between optimal estimators and convex relaxation-based estimators depend on the sparsity level s∗?
• RQ4 Can this gap be quantified without relying on unproven computational hardness assumptions?
• RQ5 Is the SoS hierarchy, as a tight convex relaxation, still statistically suboptimal compared to computationally infeasible estimators?

Key findings

• A computational infeasible estimator can achieve the information-theoretic lower bound of order √(1/s∗ · log(d/s∗)) for both sparse principal submatrix and stochastic block model estimation.
• Any estimator within the SoS hierarchy (or weaker convex relaxation hierarchies) incurs an error lower bound of order C′, independent of s∗, indicating a fundamental statistical cost.
• The gap between the optimal rate and the rate achievable via convex relaxations is of order √s∗, which is quantitatively established under the regime s∗ = o((d / log d)^(1/2ℓ)) for ℓ-th level SoS.
• For sparse principal submatrix estimation, a linear-time estimator within the SoS hierarchy achieves the lower bound up to a logarithmic factor, showing near-optimality within this class.
• The proof relies on a construction-based approach using dual certificates inspired by Meka et al. (2015), avoiding reductions from hard problems like planted clique.
• The results hold without unproven computational hardness hypotheses, distinguishing this work from prior computational lower bounds in statistics.