[论文解读] The noise barrier and the large signal bias of the Lasso and other convex estimators

Pierre Bellec|arXiv (Cornell University)|Apr 4, 2018

Statistical Methods and Inference参考文献 5被引用 10

一句话总结

本文引入了噪声屏障（noise barrier）和大信号偏差（large signal bias）作为新型诊断工具，用于解释Lasso等凸估计量的预测误差。证明了设计矩阵的相容性条件对于实现快速预测速率是不可避免的，并基于调参参数相对于临界阈值的取值，揭示了Lasso性能的尖锐相变现象，其结果对随机的、基于数据的调参参数也具有启示意义。

ABSTRACT

Convex estimators such as the Lasso, the matrix Lasso and the group Lasso have been studied extensively in the last two decades, demonstrating great success in both theory and practice. Two quantities are introduced, the noise barrier and the large scale bias, that provides insights on the performance of these convex regularized estimators. It is now well understood that the Lasso achieves fast prediction rates, provided that the correlations of the design satisfy some Restricted Eigenvalue or Compatibility condition, and provided that the tuning parameter is large enough. Using the two quantities introduced in the paper, we show that the compatibility condition on the design matrix is actually unavoidable to achieve fast prediction rates with the Lasso. The Lasso must incur a loss due to the correlations of the design matrix, measured in terms of the compatibility constant. This results holds for any design matrix, any active subset of covariates, and any tuning parameter. It is now well known that the Lasso enjoys a dimension reduction property: the prediction error is of order $λ\sqrt k$ where $k$ is the sparsity; even if the ambient dimension $p$ is much larger than $k$. Such results require that the tuning parameters is greater than some universal threshold. We characterize sharp phase transitions for the tuning parameter of the Lasso around a critical threshold dependent on $k$. If $λ$ is equal or larger than this critical threshold, the Lasso is minimax over $k$-sparse target vectors. If $λ$ is equal or smaller than critical threshold, the Lasso incurs a loss of order $σ\sqrt k$ -- which corresponds to a model of size $k$ -- even if the target vector has fewer than $k$ nonzero coefficients. Remarkably, the lower bounds obtained in the paper also apply to random, data-driven tuning parameters. The results extend to convex penalties beyond the Lasso.

研究动机与目标

• 理解Lasso等凸估计量在高维稀疏线性回归中的基本局限性。
• 识别为何设计矩阵的相容性条件对于Lasso实现快速预测速率是必要的。
• 基于调参参数相对于临界阈值的取值，刻画Lasso预测误差的尖锐相变现象。
• 将分析扩展至Lasso以外的凸惩罚项，包括核范数和组Lasso惩罚。
• 证明即使对于随机的、基于数据的调参参数，预测误差的下界依然成立。

提出的方法

• 引入两个新量：噪声屏障与大信号偏差，用于刻画凸估计量的预测误差。
• 利用噪声屏障推导预测误差的下界，表明当调参参数低于临界阈值时性能会下降。
• 确立设计矩阵的相容性常数量化了由预测变量间相关性带来的不可避免偏差。
• 将该框架应用于ℓ₁-惩罚的Lasso，并扩展至具有核范数和组Lasso惩罚的矩阵Lasso与组Lasso场景。
• 采用适用于非线性估计量的偏差-方差型分解，使预测误差可被解释为设计矩阵结构与调参参数选择的函数。
• 证明即使在满足温和矩条件的前提下，若数据驱动的调参参数期望值低于临界阈值，其预测误差仍无法避免该下界。

实验结果

研究问题

• RQ1 设计矩阵的相容性条件是否确实对Lasso实现快速预测速率是必不可少的？
• RQ2 当调参参数低于临界阈值时，Lasso的预测误差会发生什么变化？
• RQ3 噪声屏障与大信号偏差是否可用于理解Lasso以外凸估计量的性能表现？
• RQ4 对于随机的、基于数据的调参参数，预测误差的下界是否依然成立？
• RQ5 在低秩矩阵恢复中，核范数惩罚估计量的性能与无惩罚最小二乘法相比如何？

主要发现

• 设计矩阵的相容性条件对于Lasso实现快速预测速率是不可避免的；若不满足该条件，偏差将与相容性常数的倒数成正比。
• Lasso的预测误差中存在尖锐相变：若调参参数λ低于临界阈值，即使真实向量的非零系数远少于k个，误差也至少为σ√k量级。
• 对于满足温和矩条件的数据驱动调参参数，若其期望值低于临界阈值，则预测误差的下界仍为σ√k量级。
• 当期望调参参数低于cσ√m/4时，核范数惩罚估计量的预测误差下界为σ√(Tm)量级，表明此时其性能无法优于无惩罚最小二乘法。
• 大信号偏差的下界表明，即使目标向量稀疏且调参参数自适应选择，设计矩阵的相关性仍会内在地惩罚预测误差。
• 该结果可推广至一般凸惩罚项，包括组Lasso与矩阵Lasso，其下界形式类似，取决于惩罚结构与设计矩阵的特性。