[论文解读] An l1-Oracle Inequality for the Lasso

Pascal Massart, Caroline Meynet|arXiv (Cornell University)|Jul 27, 2010

Statistical Methods and Inference参考文献 17被引用 10

一句话总结

本文在不假设设计或回归函数具有几何结构的前提下，建立了Lasso估计量的 $ε_{1}$-oracle 不等式，证明当正则化参数选择得当时，其性能几乎与确定性Lasso相当。此外，通过在二进制截断上进行 $ε_{0}$-惩罚选择，提出了适用于无限词典的选定Lasso估计量，在插值空间中实现了最优收敛速率。

ABSTRACT

The Lasso has attracted the attention of many authors these last years. While many efforts have been made to prove that the Lasso behaves like a variable selection procedure at the price of strong (though unavoidable) assumptions on the geometric structure of these variables, much less attention has been paid to the analysis of the performance of the Lasso as a regularization algorithm. Our first purpose here is to provide a conceptually very simple result in this direction. We shall prove that, provided that the regularization parameter is properly chosen, the Lasso works almost as well as the deterministic Lasso. This result does not require any assumption at all, neither on the structure of the variables nor on the regression function. Our second purpose is to introduce a new estimator particularly adapted to deal with infinite countable dictionaries. This estimator is constructed as an l0-penalized estimator among a sequence of Lasso estimators associated to a dyadic sequence of growing truncated dictionaries. The selection procedure automatically chooses the best level of truncation of the dictionary so as to make the best tradeoff between approximation, l1-regularization and sparsity. From a theoretical point of view, we shall provide an oracle inequality satisfied by this selected Lasso estimator. The oracle inequalities established for the Lasso and the selected Lasso estimators shall enable us to derive rates of convergence on a wide class of functions, showing that these estimators perform at least as well as greedy algorithms. Besides, we shall prove that the rates of convergence achieved by the selected Lasso estimator are optimal in the orthonormal case by bounding from below the minimax risk on some Besov bodies. Finally, some theoretical results about the performance of the Lasso for infinite uncountable dictionaries will be studied in the specific framework of neural networks. All the oracle inequalities presented in this paper are obtained via the application of a single general theorem of model selection among a collection of nonlinear models which is a direct consequence of the Gaussian concentration inequality. The key idea that enables us to apply this general theorem is to see l1-regularization as a model selection procedure among l1-balls.

研究动机与目标

• 在不施加对设计或回归函数的限制性假设条件下，分析Lasso作为正则化方法的性能。
• 为无限可数词典构造一种新估计量，该估计量在逼近误差、$ε_{1}$-正则化与稀疏性之间实现平衡。
• 推导Lasso与选定Lasso估计量在一般函数类中的oracle不等式与收敛速率。
• 通过基于 $ε_{1}$-球的通用模型选择定理，统一分析Lasso估计量。
• 证明Lasso在收敛速率方面可达到与贪婪算法相当的性能。

提出的方法

• 将Lasso解释为在 $ε_{1}$-球族中进行模型选择的过程，从而可应用通用模型选择定理。
• 在不假设词典或回归函数结构的前提下，推导出Lasso的 $ε_{1}$-oracle 不等式。
• 通过在二进制增长的截断词典上对一系列Lasso估计量施加 $ε_{0}$-惩罚，构造选定Lasso估计量。
• 选择过程自动选取最优截断水平，以平衡逼近误差、$ε_{1}$-正则化与稀疏性。
• 利用插值空间与实插值理论推导收敛速率。
• 通过通用模型选择定理建立理论保证，关键技术步骤依赖于熵与覆盖数的界。

实验结果

研究问题

• RQ1 Lasso能否在不假设设计或回归函数具有几何结构的前提下，作为正则化方法进行分析？
• RQ2 如何为无限词典构造一个一致估计量，使其能自适应地选择最优截断水平？
• RQ3 Lasso在一般函数类中的收敛速率表现如何？
• RQ4 能否将统一的模型选择框架应用于基于 $ε_{1}$-球作为模型的 $ε_{1}$-正则化估计量？
• RQ5 Lasso与选定Lasso估计量在收敛速率方面与贪婪算法相比如何？

主要发现

• 当正则化参数选择得当时，Lasso在不假设词典或回归函数结构的前提下，满足 $ε_{1}$-oracle 不等式。
• 通过在二进制截断上进行 $ε_{0}$-惩罚选择构建的选定Lasso估计量，实现了平衡逼近误差、$ε_{1}$-正则化与稀疏性的oracle不等式。
• Lasso估计量在插值空间中的收敛速率与贪婪算法相当或更优。
• 在 $ε_{q}(R) \cap \mathcal{B}^{r}_{2,\infty}(R)$ 中推导出极小化风险的下界，证明所达到的速率具有最优性。
• 选定Lasso估计量的收敛速率为 $\kappa'' R^q (\varepsilon \sqrt{\ln(R\varepsilon^{-1})})^{2-q}$，其中 $q \in (0,2)$ 且 $\kappa'' > 0$ 为绝对常数。
• 分析表明，即使在缺乏结构假设的条件下，Lasso在风险表现上也几乎与确定性Lasso相当。