[论文解读] Post-L1-Penalized Estimators in High-Dimensional Linear Regression Models

Alexandre Belloni, Victor Chernozhukov|arXiv (Cornell University)|Dec 31, 2009

Statistical Methods and Inference被引用 1

一句话总结

本文提出了后LASSO估计量，即对LASSO所选模型应用无惩罚回归，表明其在保持LASSO收敛速度的同时减少了偏差。关键的是，只要所选模型包含所有真实成分且足够稀疏，后LASSO即使在LASSO遗漏真实预测变量时也能优于LASSO，当LASSO完美选择真实模型时，后LASSO可达到Oracle速率。

ABSTRACT

In this paper we study post-penalized estimators which apply ordinary, unpenalized linear regression to the model selected by first-step penalized estimators, typically LASSO. It is well known that LASSO can estimate the regression function at nearly the oracle rate, and is thus hard to improve upon. We show that post-LASSO performs at least as well as LASSO in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the LASSO-based model selection 'fails' in the sense of missing some components of the 'true' regression model. By the 'true' model we mean here the best s-dimensional approximation to the regression function chosen by the oracle. Furthermore, post-LASSO can perform strictly better than LASSO, in the sense of a strictly faster rate of convergence, if the LASSO-based model selection correctly includes all components of the 'true' model as a subset and also achieves a sufficient sparsity. In the extreme case, when LASSO perfectly selects the 'true' model, the post-LASSO estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by LASSO which guarantees that this dimension is at most of the same order as the dimension of the 'true' model. Our rate results are non-asymptotic and hold in both parametric and nonparametric models. Moreover, our analysis is not limited to the LASSO estimator in the first step, but also applies to other estimators, for example, the trimmed LASSO, Dantzig selector, or any other estimator with good rates and good sparsity. Our analysis covers both traditional trimming and a new practical, completely data-driven trimming scheme that induces maximal sparsity subject to maintaining a certain goodness-of-fit. The latter scheme has theoretical guarantees similar to those of LASSO or post-LASSO, but it dominates these procedures as well as traditional trimming in a wide variety of experiments.

研究动机与目标

• 分析后LASSO估计量在高维线性回归模型中的理论性能。
• 理解后LASSO在收敛速度和偏差方面优于LASSO的条件。
• 建立LASSO所选模型维度的非渐近稀疏性界。
• 将分析从LASSO扩展至其他估计量，如截断LASSO和Dantzig选择器。
• 提出并证明一种数据驱动的截断方案，以在保持拟合优度的前提下最大化稀疏性。

提出的方法

• 对第一步使用惩罚估计量（如LASSO）所选模型应用普通最小二乘法，形成后LASSO估计量。
• 推导依赖于所选模型稀疏性的估计误差的非渐近界。
• 提出一种新的稀疏性界，确保LASSO所选模型的维度与真实模型维度同阶。
• 提出一种数据驱动的截断程序，选择在指定拟合优度下最稀疏的模型，并提供理论保证。
• 通过利用其优良的收敛速率和稀疏性特征，将该框架扩展至其他估计量，包括Dantzig选择器和截断LASSO。
• 使用Oracle风险比较评估性能，表明当模型选择完美时，后LASSO可达到Oracle速率。

实验结果

研究问题

• RQ1 在何种条件下，后LASSO的收敛速度快于LASSO？
• RQ2 当LASSO未能包含真实模型的所有成分时，后LASSO是否仍能保持良好性能？
• RQ3 LASSO所选模型的稀疏性与真实模型维度有何关系？
• RQ4 一种在拟合优度约束下最大化稀疏性的数据驱动截断方案的理论性质为何？
• RQ5 后LASSO框架能否推广至LASSO以外的估计量，如Dantzig选择器？

主要发现

• 后LASSO与LASSO具有相同的收敛速度，但偏差更小，因此估计更准确。
• 当LASSO所选模型包含所有真实成分且足够稀疏时，后LASSO可严格优于LASSO。
• 当LASSO完美选择真实模型时，后LASSO即为Oracle估计量，达到最优收敛速率。
• 新提出的稀疏性界确保LASSO所选模型的维度不超过真实模型维度的同阶数量级。
• 所提出的基于数据的截断方案在理论上与LASSO和后LASSO性能相当，且在实证实验中表现更优。
• 该框架可普遍适用于其他具有优良收敛速率和稀疏性的估计量，包括Dantzig选择器和截断LASSO。