[Paper Review] Fundamental Barriers to High-Dimensional Regression with Convex Penalties

Michael Celentano, Andrea Montanari|arXiv (Cornell University)|Mar 25, 2019

Statistical Methods and Inference4 citations

TL;DR

This paper identifies fundamental statistical limitations of convex regularization in high-dimensional regression, showing that when the true parameter vector's coordinate distribution is non-log-concave, convex methods suffer from unavoidable performance gaps relative to optimal estimators and algorithms—even under high signal-to-noise ratios. The key result is that such gaps emerge precisely when the prior on β₀ is non-log-concave, establishing a sharp theoretical boundary for convex methods' statistical efficiency.

ABSTRACT

In high-dimensional regression, we attempt to estimate a parameter vector $β_0\in\mathbb{R}^p$ from $n\lesssim p$ observations $\{(y_i,x_i)\}_{i\leq n}$ where $x_i\in\mathbb{R}^p$ is a vector of predictors and $y_i$ is a response variable. A well-established approach uses convex regularizers to promote specific structures (e.g. sparsity) of the estimate $\widehatβ$, while allowing for practical algorithms. Theoretical analysis implies that convex penalization schemes have nearly optimal estimation properties in certain settings. However, in general the gaps between statistically optimal estimation (with unbounded computational resources) and convex methods are poorly understood. We show that when the statistican has very simple structural information about the distribution of the entries of $β_0$, a large gap frequently exists between the best performance achieved by any convex regularizer satisfying a mild technical condition and either (i) the optimal statistical error or (ii) the statistical error achieved by optimal approximate message passing algorithms. Remarkably, a gap occurs at high enough signal-to-noise ratio if and only if the distribution of the coordinates of $β_0$ is not log-concave. These conclusions follow from an analysis of standard Gaussian designs. Our lower bounds are expected to be generally tight, and we prove tightness under certain conditions.

Motivation & Objective

• To understand the fundamental statistical limitations of convex regularization in high-dimensional regression, especially when computational efficiency is required.
• To investigate the gap between the best possible performance of convex M-estimators and the optimal statistical error (Bayes risk) or the performance of optimal approximate message passing (AMP) algorithms.
• To determine under what conditions—particularly in terms of the distribution of β₀'s coordinates—convex methods fail to achieve optimal statistical performance.
• To establish that non-log-concavity of the prior distribution on β₀ is both necessary and sufficient for the existence of such performance gaps.

Proposed method

• The authors analyze the asymptotic performance of convex M-estimators under standard Gaussian random design matrices using tools from high-dimensional asymptotic statistics and state evolution of approximate message passing (AMP).
• They introduce a convex lower bound on the estimation error that characterizes the best possible performance achievable by any convex regularizer satisfying a mild technical condition.
• The analysis compares this convex lower bound to the risk of Bayes-AMP (optimal polynomial-time algorithm) and the Bayes risk (information-theoretic lower bound), using asymptotic fixed-point equations.
• The authors prove tightness of their lower bounds under certain conditions, particularly for δ > 1 (where δ = n/p), and use regularity lemmas and pseudo-Lipschitz function properties to control concentration and convergence.
• They employ Tweedie’s formula and Stein’s lemma to analyze conditional expectations in high-dimensional Gaussian models, enabling exact asymptotic characterization of estimation error.
• The framework is validated through examples involving sparsity-inducing penalties (e.g., ℓ₁, SLOPE, OWL norms), and the results are shown to be robust across different convex regularizers.

Experimental results

Research questions

• RQ1 Under what conditions does a fundamental gap exist between the performance of convex M-estimators and the optimal statistical error (Bayes risk) in high-dimensional regression?
• RQ2 Is there a performance gap between convex M-estimators and optimal approximate message passing (AMP) algorithms, and if so, under what distributional assumptions on β₀?
• RQ3 Why do convex methods fail to achieve optimal statistical performance when the distribution of β₀'s coordinates is non-log-concave, and what is the precise role of log-concavity in this context?
• RQ4 Can the convex lower bound on estimation error be proven tight, and under what conditions does it match the actual performance of convex estimators?
• RQ5 How does the signal-to-noise ratio (SNR) influence the existence and magnitude of the gap between convex methods and optimal benchmarks?

Key findings

• A fundamental gap exists between the best convex M-estimator and the optimal statistical error (Bayes risk) if and only if the distribution of the coordinates of β₀ is not log-concave.
• At high signal-to-noise ratio (SNR), such a gap appears precisely when the prior on β₀ is non-log-concave, indicating a sharp statistical-computational trade-off.
• The gap between convex M-estimators and the optimal AMP algorithm also emerges under the same non-log-concave condition, showing that convexity limits algorithmic performance beyond just statistical efficiency.
• The convex lower bound on estimation error is tight under certain conditions, particularly when δ > 1 (n/p > 1), and the asymptotic performance is characterized via fixed-point equations in the state evolution framework.
• For non-log-concave priors such as those inducing sparsity (e.g., Laplace, point masses), convex penalties like ℓ₁-norm fail to achieve the optimal estimation error, even with optimal tuning.
• The results are general and apply to a broad class of convex penalties, including separable, strongly convex, and constrained regularizers, showing that the gap is structural rather than method-specific.