[論文レビュー] The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning

Léo Miolane, Andrea Montanari|arXiv (Cornell University)|Nov 3, 2018

Statistical Methods and Inference参考文献 60被引用数 55

ひとこと要約

この論文はランダムガウス設計の下でLassoに対する uniform 高確率集中結果を示し、ell_pボールと正則化に対して一様性を持ち、これを用いて適応的な調整手順を正当化する。

ABSTRACT

The Lasso is a popular regression method for high-dimensional problems in which the number of parameters $\ heta_1,\\dots,\ heta_N$, is larger than the number $n$ of samples: $N>n$. A useful heuristics relates the statistical properties of the Lasso estimator to that of a simple soft-thresholding denoiser,in a denoising problem in which the parameters $(\ heta_i)_{i\\le N}$ are observed in Gaussian noise, with a carefully tuned variance. Earlier work confirmed this picture in the limit $n,N\ o\\infty$, pointwise in the parameters $\ heta$, and in the value of the regularization parameter. Here, we consider a standard random design model and prove exponential concentration of its empirical distribution around the prediction provided by the Gaussian denoising model. Crucially, our results are uniform with respect to $\ heta$ belonging to $\\ell_q$ balls, $q\\in [0,1]$, and with respect to the regularization parameter. This allows to derive sharp results for the performances of various data-driven procedures to tune the regularization. Our proofs make use of Gaussian comparison inequalities, and in particular of a version of Gordon's minimax theorem developed by Thrampoulidis, Oymak, and Hassibi, which controls the optimum value of the Lasso optimization problem. Crucially, we prove a stability property of the minimizer in Wasserstein distance, that allows to characterize properties of the minimizer itself.

研究の動機と目的

• 標準的なランダム設計の下で、Lassoの経験分布がガウスデノイザー予測の周りにどのように集中するかを動機づけ、定量化する。
• 楕円球ℓ_pボールとλに対してパラメータに一様な結果を提供し、正則化パラメータのデータ駆動チューニングを可能にする。
• デビアイズドLassoの分布を特徴づけ、ワッサースタイン距離で最適解の振る舞いを推定する安定性性質を確立する。
• 一様なリスクとノイズレベル推定量を開発し、適応的なλ選択への利用を示す。
• EST、SURE、クロスバリデーションのような適応手順を裏付け、境界付けする方法を示す。
• Lasso最適化のスカラー極限等価を介して理論をミニマックスの考慮に結びつける。

提案手法

• モデル: Xがガウス設計でノイズzを用いた線形回帰; y = Xθ⋆ + σz, ただし Xij ~ N(0,1/n)。
• Lasso推定量: θ̂λ = argminθ (1/2n)||y − Xθ||^2 + (λ/n)||θ||1。
• 主要解析手段: ガウス比較不等式（Gordonのミニマックス定理）と、最小値を最小化解に結びつけるワッサースタイン距離の安定性性質。
• 固定点方程式（5）と関連量（τ*, α*）は、デビアイズドおよび通常のLasso推定量の漸近的分布を特徴づける。
• 一様収束の結果（定理3.1）は、経験分布が μλ* に一様に収束することを示す。θ⋆がℓp-ボール内、λが[λmin, λmax]の範囲で。
• リスク R*(λ)、予測 P*(λ) の定義とそれらの一様推定（系説4.1–4.4）。
• 適応正則化チューニングへの結果の適用: 保証付き EST、SURE、k分割CV（命題4.1–4.3）。
• デビアイズドLassoの分布（定理3.3）と μ(λ) d へのワッサースタイン収束。

実験結果

リサーチクエスチョン

• RQ1 Does the Lasso under Gaussian design exhibit uniform concentration of its empirical distribution around the Gaussian-denoising model across λ and θ in ℓp-balls?
• RQ2 Can one derive uniform (in λ and θ) laws to support data-driven tuning of the regularization parameter using adaptive procedures (EST, SURE, CV)?
• RQ3 What are the risk, noise level, and prediction-error estimators that remain consistent uniformly over sparse parameter sets?
• RQ4 How does the debiased Lasso behave under uniform control, and can its distribution be characterized in a way that supports confidence interval construction?
• RQ5 What is the role of a Wasserstein-stability property in transferring information from the Lasso minimum to the estimator itself?

主な発見

• Empirical distribution of (θ̂λ, θ⋆) concentrates around μλ* with high probability, uniformly over λ in [λmin, λmax] and θ⋆ in ℓp-balls.
• A unique fixed-point pair (β*(λ), τ*(λ)) solves the max-min problem (8), determining the asymptotic debiased distribution and related quantities.
• Uniformly consistent estimators for τ*(λ), the Lasso risk R*(λ), and prediction error, enabling reliable adaptive tuning.
• Debiased Lasso θ̂d,λ is approximately distributed as N(θ⋆, τ*^2 I) with Wasserstein concentration to μλ* (Theorem 3.3).
• Three data-driven λ selection methods (EST, SURE, CV) achieve near-optimal risk in simulations and are supported by uniform theory (Propositions 4.1–4.3).
• SURE-based and cross-validation-based estimators provide uniform consistency guarantees for the prediction error and risk estimates.