[Paper Review] The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning

Léo Miolane, Andrea Montanari|arXiv (Cornell University)|Nov 3, 2018

Statistical Methods and Inference60 references55 citations

TL;DR

The paper proves uniform, high-probability concentration results for the Lasso under random Gaussian design, uniform over ell_p balls and regularization, and uses these to justify adaptive tuning procedures.

ABSTRACT

The Lasso is a popular regression method for high-dimensional problems in which the number of parameters $\ heta_1,\\dots,\ heta_N$, is larger than the number $n$ of samples: $N>n$. A useful heuristics relates the statistical properties of the Lasso estimator to that of a simple soft-thresholding denoiser,in a denoising problem in which the parameters $(\ heta_i)_{i\\le N}$ are observed in Gaussian noise, with a carefully tuned variance. Earlier work confirmed this picture in the limit $n,N\ o\\infty$, pointwise in the parameters $\ heta$, and in the value of the regularization parameter. Here, we consider a standard random design model and prove exponential concentration of its empirical distribution around the prediction provided by the Gaussian denoising model. Crucially, our results are uniform with respect to $\ heta$ belonging to $\\ell_q$ balls, $q\\in [0,1]$, and with respect to the regularization parameter. This allows to derive sharp results for the performances of various data-driven procedures to tune the regularization. Our proofs make use of Gaussian comparison inequalities, and in particular of a version of Gordon's minimax theorem developed by Thrampoulidis, Oymak, and Hassibi, which controls the optimum value of the Lasso optimization problem. Crucially, we prove a stability property of the minimizer in Wasserstein distance, that allows to characterize properties of the minimizer itself.

Motivation & Objective

• Motivate and quantify how the Lasso's empirical distribution concentrates around a Gaussian denoiser prediction under a standard random design.
• Provide uniform-in-parameters results (over ell_p balls and lambda) to enable data-driven tuning of the regularization parameter.
• Characterize the debiased Lasso distribution and establish a stability property to infer minimizer behavior in Wasserstein distance.
• Develop uniform risk and noise level estimators and demonstrate their use for adaptive lambda selection.
• Show how the results support and bound adaptive procedures like EST, SURE, and cross-validation.
• Connect the theory to minimax considerations via a scalar-limit equivalent of the Lasso optimization.

Proposed method

• Model: linear regression with Gaussian design X and noise z; y = Xθ⋆ + σz, with Xij ~ N(0,1/n).
• Lasso estimator: θ̂λ = argminθ (1/2n)||y − Xθ||^2 + (λ/n)||θ||1.
• Key analytic tool: Gaussian comparison inequalities (Gordon’s minimax theorem) and a stability property in Wasserstein distance linking the minimum to the minimizer.
• Fixed-point equations (5) and associated quantities (τ*, α*) characterize the asymptotic law of debiased and ordinary Lasso estimators.
• Uniform convergence results (Theorem 3.1) show empirical distributions converge to μλ* uniformly over θ⋆ in ℓp-balls and λ in [λmin, λmax].
• Definitions of risk R*(λ), prediction P*(λ) and their uniform estimates (Corollaries 4.1–4.4).
• Application of results to adaptive regularization tuning: EST, SURE, and k-fold CV with guarantees (Propositions 4.1–4.3).
• Debiased Lasso distribution (Theorem 3.3) and its Wasserstein convergence to μ(λ) d.

Experimental results

Research questions

• RQ1 Does the Lasso under Gaussian design exhibit uniform concentration of its empirical distribution around the Gaussian-denoising model across λ and θ in ℓp-balls?
• RQ2 Can one derive uniform (in λ and θ) laws to support data-driven tuning of the regularization parameter using adaptive procedures (EST, SURE, CV)?
• RQ3 What are the risk, noise level, and prediction-error estimators that remain consistent uniformly over sparse parameter sets?
• RQ4 How does the debiased Lasso behave under uniform control, and can its distribution be characterized in a way that supports confidence interval construction?
• RQ5 What is the role of a Wasserstein-stability property in transferring information from the Lasso minimum to the estimator itself?

Key findings

• Empirical distribution of (θ̂λ, θ⋆) concentrates around μλ* with high probability, uniformly over λ in [λmin, λmax] and θ⋆ in ℓp-balls.
• A unique fixed-point pair (β*(λ), τ*(λ)) solves the max-min problem (8), determining the asymptotic debiased distribution and related quantities.
• Uniformly consistent estimators for τ*(λ), the Lasso risk R*(λ), and prediction error, enabling reliable adaptive tuning.
• Debiased Lasso θ̂d,λ is approximately distributed as N(θ⋆, τ*^2 I) with Wasserstein concentration to μλ* (Theorem 3.3).
• Three data-driven λ selection methods (EST, SURE, CV) achieve near-optimal risk in simulations and are supported by uniform theory (Propositions 4.1–4.3).
• SURE-based and cross-validation-based estimators provide uniform consistency guarantees for the prediction error and risk estimates.