[Paper Review] Optimistic lower bounds for convex regularized least-squares

Pierre Bellec|arXiv (Cornell University)|Mar 3, 2017

Sparse and Compressive Sensing Techniques23 references6 citations

TL;DR

This paper introduces 'optimistic lower bounds' for convex regularized least-squares estimators, providing prediction error bounds that apply to any target vector—not just worst-case scenarios. By characterizing the prediction error via variational functions F, G, and H, the framework yields tight, data-dependent lower and upper bounds, with applications to the Lasso showing matching bounds for universal tuning and new lower bounds for small tuning parameters.

ABSTRACT

Minimax lower bounds are pessimistic in nature: for any given estimator, minimax lower bounds yield the existence of a worst-case target vector $β^*_{worst}$ for which the prediction error of the given estimator is bounded from below. However, minimax lower bounds shed no light on the prediction error of the given estimator for target vectors different than $β^*_{worst}$. A characterization of the prediction error of any convex regularized least-squares is given. This characterization provide both a lower bound and an upper bound on the prediction error. This produces lower bounds that are applicable for any target vector and not only for a single, worst-case $β^*_{worst}$. Finally, these lower and upper bounds on the prediction error are applied to the Lasso is sparse linear regression. We obtain a lower bound involving the compatibility constant for any tuning parameter, matching upper and lower bounds for the universal choice of the tuning parameter, and a lower bound for the Lasso with small tuning parameter.

Motivation & Objective

• To overcome the limitations of minimax lower bounds, which only characterize worst-case performance, by developing bounds applicable to any target vector.
• To provide a unified variational framework for characterizing the prediction error of convex regularized least-squares estimators.
• To derive tight lower and upper bounds on prediction error that are valid for all target vectors, not just worst-case ones.
• To apply the framework to the Lasso in sparse linear regression, yielding new insights into its performance across tuning parameter regimes.
• To establish concentration properties of the prediction error under Gaussian noise, enabling probabilistic guarantees.

Proposed method

• Introduces a variational characterization of the prediction error using the function $ F(t) = \sup_{\|\mathbf{X}(\beta - \beta^*)\| \leq t} \left( \varepsilon^T \mathbf{X}(\beta - \beta^*) - h(\beta) \right) - t^2/2 $, which links the estimator’s error to a maximization problem.
• Establishes that the true prediction error $ \|\mathbf{X}(\hat{\beta} - \beta^*)\| $ is almost surely a maximizer of $ F(t) $, enabling the use of optimization tools.
• Defines auxiliary functions $ G $ and $ H $ to derive upper and lower bounds on the prediction error, leveraging strong concavity and Lipschitz properties.
• Uses concentration inequalities for Lipschitz functions of Gaussian noise to derive probabilistic bounds on the prediction error, particularly under standard normal noise.
• Applies the framework to the Lasso by analyzing the compatibility constant and deriving bounds that match known upper bounds for the universal tuning parameter choice.
• Employs a median-based argument and union bounds to show that the event where the prediction error is close to its median has positive probability, enabling non-asymptotic guarantees.

Experimental results

Research questions

• RQ1 Can we derive lower bounds on the prediction error of convex regularized least-squares that are valid for any target vector, not just the worst-case one?
• RQ2 How can we characterize the prediction error of a convex regularized estimator using variational functions that depend on the noise and design matrix?
• RQ3 What are the implications of these optimistic bounds for the Lasso in sparse linear regression, particularly for small and universal tuning parameters?
• RQ4 Can we establish concentration properties of the prediction error when the noise is standard normal, using the proposed framework?
• RQ5 How do the proposed optimistic bounds compare to existing minimax lower bounds in terms of tightness and applicability?

Key findings

• The prediction error $ \|\mathbf{X}(\hat{\beta} - \beta^*)\| $ is almost surely a maximizer of the random function $ F(t) $, enabling a variational characterization of the error.
• For any target vector $ \beta^* $, the framework provides both a lower and an upper bound on the prediction error, improving upon minimax bounds that only consider worst-case $ \beta^*_{\text{worst}} $.
• For the Lasso with the universal tuning parameter, the optimistic lower bound matches the known upper bound, indicating tightness of the bound.
• The paper derives a new lower bound for the Lasso with small tuning parameters, which is non-vacuous and applicable beyond worst-case scenarios.
• Under Gaussian noise, the prediction error concentrates around its median, and the framework yields a high-probability bound of the form $ |\sqrt{m} - \sqrt{t_f}| \leq \sqrt{21\sigma/2} $ with positive probability.
• The framework improves upon prior results, such as Proposition 1.3 in Chatterjee (2014), by reducing the constant factor in the concentration bound from 1 to 1/2 in the relevant inequality.