[Paper Review] Sharp thresholds for high-dimensional and noisy recovery of sparsity

Martin J. Wainwright|ArXiv.org|May 30, 2006

Sparse and Compressive Sensing TechniquesEngineering32 references152 citations

TL;DR

This paper establishes sharp thresholds for successful sparsity pattern recovery using the Lasso (ℓ₁-constrained quadratic programming) in high-dimensional, noisy settings. It proves that for Gaussian design matrices, exact recovery is possible with high probability when the sample size n exceeds approximately 2(log p) + s, with a precise threshold at θ = 1, making the phase transition sharp and exactly characterized.

ABSTRACT

The problem of consistently estimating the sparsity pattern of a vector $\betastar \in eal^\mdim$ based on observations contaminated by noise arises in various contexts, including subset selection in regression, structure estimation in graphical models, sparse approximation, and signal denoising. We analyze the behavior of $\ell_1$-constrained quadratic programming (QP), also referred to as the Lasso, for recovering the sparsity pattern. Our main result is to establish a sharp relation between the problem dimension $\mdim$, the number $\spindex$ of non-zero elements in $\betastar$, and the number of observations $ umobs$ that are required for reliable recovery. For a broad class of Gaussian ensembles satisfying mutual incoherence conditions, we establish existence and compute explicit values of thresholds $\ThreshLow$ and $\ThreshUp$ with the following properties: for any $ε> 0$, if $ umobs > 2 (\ThreshUp + ε) \log (\mdim - \spindex) + \spindex + 1$, then the Lasso succeeds in recovering the sparsity pattern with probability converging to one for large problems, whereas for $ umobs < 2 (\ThreshLow - ε) \log (\mdim - \spindex) + \spindex + 1$, then the probability of successful recovery converges to zero. For the special case of the uniform Gaussian ensemble, we show that $\ThreshLow = \ThreshUp = 1$, so that the threshold is sharp and exactly determined.

Motivation & Objective

• To determine the precise conditions under which the Lasso can consistently recover the sparsity pattern of a high-dimensional, sparse vector β* from noisy observations.
• To establish sharp thresholds in terms of sample size n, dimension p, and sparsity s for successful support recovery.
• To analyze the behavior of the Lasso under general Gaussian random design ensembles with mutual incoherence conditions.
• To derive exact, non-asymptotic conditions under which the probability of correct sparsity pattern recovery converges to one or zero.
• To show that for the uniform Gaussian ensemble, the threshold is sharp and exactly θ = 1, providing a precise phase transition.

Proposed method

• Analyzes the Lasso via ℓ₁-constrained quadratic programming: minimize (1/(2n))||Y - Xβ||² + λ||β||₁.
• Uses random matrix theory and extreme value theory for Gaussian processes to bound the maximum correlation between noise and inactive predictors.
• Derives lower and upper bounds on the expected maximum of Gaussian processes to characterize the separation between active and inactive variables.
• Applies concentration inequalities and asymptotic results on extrema of i.i.d. Gaussian sequences to control the behavior of the dual certificate.
• Introduces a dual certificate construction based on the inverse covariance of the design matrix to verify support recovery.
• Employs mutual incoherence conditions and spectral properties of the design matrix to derive non-asymptotic threshold conditions.

Experimental results

Research questions

• RQ1 What is the precise threshold sample size n required for the Lasso to recover the true sparsity pattern with high probability in high-dimensional, noisy settings?
• RQ2 How do the dimension p, sparsity s, and sample size n interact to determine the success or failure of sparsity recovery?
• RQ3 Is the phase transition for Lasso support recovery sharp, and if so, can the threshold be exactly computed?
• RQ4 What is the behavior of the Lasso under the uniform Gaussian ensemble, and does it achieve a sharp threshold?
• RQ5 How do mutual incoherence and spectral properties of the design matrix affect the recovery threshold?

Key findings

• For a broad class of Gaussian ensembles satisfying mutual incoherence, there exist sharp thresholds θℓ and θu such that if n > 2(θu + ν)log(p−s) + s + 1, recovery succeeds with high probability.
• If n < 2(θℓ − ν)log(p−s) + s + 1, the probability of successful recovery converges to zero.
• For the uniform Gaussian ensemble (i.e., X_k ~ N(0, I_p)), the thresholds coincide: θℓ = θu = 1, resulting in a sharp, exact threshold.
• The threshold condition is n > 2log(p−s) + s + 1 for reliable recovery, with convergence to probability one as problem size increases.
• The analysis confirms that the Lasso achieves consistent sparsity pattern recovery under the specified conditions, even when p ≫ n.
• The dual certificate construction and extreme value analysis of Gaussian processes are critical in deriving the exact threshold and proving the sharp phase transition.