[论文解读] CAD: Debiasing the Lasso with inaccurate covariate model

Michael Celentano, Andrea Montanari|arXiv (Cornell University)|Jul 29, 2021

Statistical Methods and Inference参考文献 47被引用 4

一句话总结

本文提出相关性校正去偏Lasso（CAD），一种新颖方法，用于在协变量模型估计不准确时，构建高维线性回归中近似无偏的估计量。CAD通过校正精度矩阵与回归参数估计误差之间的相关性，消除偏差，即使在非主成分协方差估计不佳的情况下，也能在联合高斯协变量的半监督设置下实现近乎完美的偏差抵消。

ABSTRACT

We consider the problem of estimating a low-dimensional parameter in high-dimensional linear regression. Constructing an approximately unbiased estimate of the parameter of interest is a crucial step towards performing statistical inference. Several authors suggest to orthogonalize both the variable of interest and the outcome with respect to the nuisance variables, and then regress the residual outcome with respect to the residual variable. This is possible if the covariance structure of the regressors is perfectly known, or is sufficiently structured that it can be estimated accurately from data (e.g., the precision matrix is sufficiently sparse). Here we consider a regime in which the covariate model can only be estimated inaccurately, and hence existing debiasing approaches are not guaranteed to work. When errors in estimating the covariate model are correlated with errors in estimating the linear model parameter, an incomplete elimination of the bias occurs. We propose the Correlation Adjusted Debiased Lasso (CAD), which nearly eliminates this bias in some cases, including cases in which the estimation errors are neither negligible nor orthogonal. We consider a setting in which some unlabeled samples might be available to the statistician alongside labeled ones (semi-supervised learning), and our guarantees hold under the assumption of jointly Gaussian covariates. The new debiased estimator is guaranteed to cancel the bias in two cases: (1) when the total number of samples (labeled and unlabeled) is larger than the number of parameters, or (2) when the covariance of the nuisance (but not the effect of the nuisance on the variable of interest) is known. Neither of these cases is treated by state-of-the-art methods.

研究动机与目标

• 解决在协变量模型存在误差时，构建高维线性回归中低维参数近似无偏估计量的挑战。
• 克服现有去偏方法在精度矩阵与回归系数估计误差相关时失效的问题。
• 开发一种即使在非主成分协方差结构估计不准确时也能确保偏差抵消的方法。
• 为半监督学习设置下（含未标记数据）的去偏提供理论保证。
• 将去偏Lasso方法的适用范围扩展至无需精确或稀疏精度矩阵估计的场景。

提出的方法

• 该方法引入一种相关性校正校正项，以考虑精度矩阵与回归参数估计误差之间的协方差。
• 利用标记数据与未标记数据联合改进非主成分协方差结构的估计。
• 通过将感兴趣变量与结果变量相对于非主成分变量正交化，随后进行校正回归步骤来构建估计量。
• 在校正项的推导中假设协变量为联合高斯分布，从而实现对偏差结构的解析控制。
• 当总样本量（标记+未标记）超过参数个数，或当非主成分协方差已知时，该方法可实现偏差抵消。
• 理论分析基于高维渐近理论，并在高斯假设下使用集中不等式。

实验结果

研究问题

• RQ1 我们能否构建一种去偏Lasso估计量，使其在精度矩阵估计不准确且其误差与回归参数误差相关时仍保持有效性？
• RQ2 在何种条件下，现有去偏方法的偏差会因估计误差的关联性而失效？
• RQ3 在模型误设下，利用未标记数据的半监督学习能否提升去偏Lasso估计量的鲁棒性？
• RQ4 当非主成分协方差未知但存在估计误差时，是否可能在高维回归中实现近乎零偏差？
• RQ5 当总样本量（标记与未标记）超过参数个数时，能否为去偏估计提供理论保证？

主要发现

• 相关性校正去偏Lasso（CAD）即使在精度矩阵与回归参数的估计误差相关时，也能在高维线性回归中近乎完全消除偏差。
• 当总样本量（标记与未标记）超过参数个数时，CAD实现了偏差抵消，这一情形未被先前方法覆盖。
• 当非主成分协方差未知但存在估计误差时，只要总样本量足够大，CAD仍保持有效性。
• 在协变量联合高斯分布的假设下，该方法可实现有效推断，从而实现对偏差结构的解析控制。
• 在高维渐近框架下建立了理论保证，表明CAD可实现估计量的渐近正态性。
• 在精度矩阵估计不准确或误差相关性较强的情境下，CAD在半监督学习场景中显著优于标准去偏Lasso。