[Paper Review] Lower bounds on the performance of polynomial-time algorithms for sparse linear regression

Yuchen Zhang, Martin J. Wainwright+1 more|arXiv (Cornell University)|Feb 9, 2014

Sparse and Compressive Sensing TechniquesEngineering20 references91 citations

TL;DR

This paper establishes a fundamental performance gap between polynomial-time algorithms and optimal methods in sparse linear regression under the standard complexity assumption NP ⊈ P/poly. It shows that when the design matrix is ill-conditioned, polynomial-time algorithms incur significantly higher minimax prediction risk than optimal algorithms, providing the first worst-case complexity-based gap without relying on average-case conjectures.

ABSTRACT

Under a standard assumption in complexity theory (NP not in P/poly), we demonstrate a gap between the minimax prediction risk for sparse linear regression that can be achieved by polynomial-time algorithms, and that achieved by optimal algorithms. In particular, when the design matrix is ill-conditioned, the minimax prediction loss achievable by polynomial-time algorithms can be substantially greater than that of an optimal algorithm. This result is the first known gap between polynomial and optimal algorithms for sparse linear regression, and does not depend on conjectures in average-case complexity.

Motivation & Objective

• To identify a fundamental performance gap between polynomial-time algorithms and optimal methods in high-dimensional sparse linear regression.
• To establish this gap under a standard worst-case complexity assumption (NP ⊈ P/poly), avoiding reliance on average-case complexity conjectures.
• To analyze the minimax prediction risk of polynomial-time algorithms in comparison to the optimal achievable risk, particularly when the design matrix is ill-conditioned.
• To demonstrate that the restricted eigenvalue condition, commonly assumed for Lasso-type methods, is insufficient to close the gap in ill-conditioned settings.
• To provide a theoretical foundation showing that computational efficiency comes at a statistical cost in certain ill-conditioned sparse regression problems.

Proposed method

• The authors analyze the minimax prediction risk for sparse linear regression using a uniform risk criterion over all k-sparse regression vectors.
• They compare the performance of the optimal ℓ₀-based estimator (computationally intractable) with that of polynomial-time algorithms, particularly ℓ₁-relaxation methods like Lasso.
• A key technical tool is the construction of a specific design matrix X with poor conditioning, which amplifies the gap between optimal and polynomial-time performance.
• The proof relies on worst-case complexity theory, specifically assuming NP ⊈ P/poly, to establish that no polynomial-time algorithm can achieve the optimal minimax risk in the ill-conditioned regime.
• They use concentration inequalities for Gaussian random matrices to bound singular values and derive lower bounds on the prediction error under the design matrix's ill-conditioning.
• The analysis includes a comparison of estimation errors via a carefully constructed perturbation argument, showing that polynomial-time estimators can suffer from inflated error in critical coordinate pairs.

Experimental results

Research questions

• RQ1 Can a provable performance gap exist between polynomial-time algorithms and optimal methods in sparse linear regression under standard worst-case complexity assumptions?
• RQ2 Does the minimax prediction risk of polynomial-time algorithms become substantially worse than the optimal risk when the design matrix is ill-conditioned?
• RQ3 Is this gap dependent on average-case complexity conjectures, or can it be established under worst-case complexity assumptions?
• RQ4 To what extent does the restricted eigenvalue condition fail to ensure optimal performance for polynomial-time methods in ill-conditioned settings?
• RQ5 Can the fundamental limits of computationally efficient sparse regression be characterized independently of statistical estimation bounds?

Key findings

• Under the assumption NP ⊈ P/poly, there exists a fundamental gap between the minimax prediction risk achievable by polynomial-time algorithms and that of optimal algorithms in sparse linear regression.
• When the design matrix is ill-conditioned, the minimax prediction loss for polynomial-time algorithms can be substantially greater than that of the optimal ℓ₀-based estimator.
• The gap is not due to statistical limitations but arises from computational intractability, as the ℓ₀-estimator is minimax-optimal but NP-hard to compute.
• The result holds without relying on average-case complexity conjectures, distinguishing it from prior work on sparse PCA and matrix detection.
• The lower bound on prediction risk for polynomial-time algorithms scales with the inverse of the restricted eigenvalue of the design matrix, which deteriorates under ill-conditioning.
• Even under the restricted eigenvalue condition, polynomial-time methods cannot achieve the optimal risk in ill-conditioned settings, indicating a fundamental computational-statistical trade-off.