[Paper Review] CAD: Debiasing the Lasso with inaccurate covariate model

Michael Celentano, Andrea Montanari|arXiv (Cornell University)|Jul 29, 2021

Statistical Methods and Inference47 references4 citations

TL;DR

This paper proposes the Correlation Adjusted Debiased Lasso (CAD), a novel method for constructing approximately unbiased estimators in high-dimensional linear regression when the covariate model is estimated inaccurately. CAD corrects for bias arising from correlated estimation errors in the precision matrix and the regression parameter, achieving near-perfect bias cancellation under semi-supervised settings with jointly Gaussian covariates, even when the nuisance covariance is poorly estimated.

ABSTRACT

We consider the problem of estimating a low-dimensional parameter in high-dimensional linear regression. Constructing an approximately unbiased estimate of the parameter of interest is a crucial step towards performing statistical inference. Several authors suggest to orthogonalize both the variable of interest and the outcome with respect to the nuisance variables, and then regress the residual outcome with respect to the residual variable. This is possible if the covariance structure of the regressors is perfectly known, or is sufficiently structured that it can be estimated accurately from data (e.g., the precision matrix is sufficiently sparse). Here we consider a regime in which the covariate model can only be estimated inaccurately, and hence existing debiasing approaches are not guaranteed to work. When errors in estimating the covariate model are correlated with errors in estimating the linear model parameter, an incomplete elimination of the bias occurs. We propose the Correlation Adjusted Debiased Lasso (CAD), which nearly eliminates this bias in some cases, including cases in which the estimation errors are neither negligible nor orthogonal. We consider a setting in which some unlabeled samples might be available to the statistician alongside labeled ones (semi-supervised learning), and our guarantees hold under the assumption of jointly Gaussian covariates. The new debiased estimator is guaranteed to cancel the bias in two cases: (1) when the total number of samples (labeled and unlabeled) is larger than the number of parameters, or (2) when the covariance of the nuisance (but not the effect of the nuisance on the variable of interest) is known. Neither of these cases is treated by state-of-the-art methods.

Motivation & Objective

• To address the challenge of constructing approximately unbiased estimators for low-dimensional parameters in high-dimensional linear regression when the covariate model is estimated with error.
• To overcome the failure of existing debiasing methods when estimation errors in the precision matrix and regression coefficients are correlated.
• To develop a method that ensures bias cancellation even when the nuisance covariance structure is inaccurately estimated.
• To provide theoretical guarantees for debiasing under semi-supervised learning settings with unlabeled data.
• To extend the applicability of debiased Lasso methods beyond settings requiring accurate or sparse precision matrix estimation.

Proposed method

• The method introduces a correlation-adjusted correction term that accounts for the covariance between estimation errors in the precision matrix and the regression parameter.
• It leverages both labeled and unlabeled data to improve the estimation of the nuisance covariance structure.
• The estimator is constructed by orthogonalizing the variable of interest and the outcome with respect to the nuisance variables, followed by a corrected regression step.
• The correction term is derived under the assumption of jointly Gaussian covariates, enabling analytical control over the bias structure.
• The method achieves bias cancellation when the total sample size (labeled + unlabeled) exceeds the number of parameters, or when the nuisance covariance is known.
• Theoretical analysis relies on high-dimensional asymptotics and concentration inequalities under Gaussianity.

Experimental results

Research questions

• RQ1 Can we construct a debiased Lasso estimator that remains valid when the precision matrix is estimated inaccurately and its errors are correlated with the regression parameter error?
• RQ2 Under what conditions does the bias in existing debiasing methods fail due to correlated estimation errors?
• RQ3 Can semi-supervised learning with unlabeled data improve the robustness of debiased Lasso estimators under model misspecification?
• RQ4 Is it possible to achieve near-zero bias in high-dimensional regression when the nuisance covariance is unknown but estimated with error?
• RQ5 What theoretical guarantees can be provided for debiased estimation when the total sample size (labeled and unlabeled) exceeds the number of parameters?

Key findings

• The Correlation Adjusted Debiased Lasso (CAD) nearly eliminates bias in high-dimensional linear regression even when estimation errors in the precision matrix and regression parameter are correlated.
• CAD achieves bias cancellation when the total number of samples (labeled and unlabeled) exceeds the number of parameters, a regime not covered by prior methods.
• CAD remains valid when the nuisance covariance is unknown but estimated with error, provided the total sample size is sufficiently large.
• The method provides valid inference under the assumption of jointly Gaussian covariates, which enables analytical control over the bias structure.
• Theoretical guarantees are established under high-dimensional asymptotics, showing that CAD achieves asymptotic normality of the estimator.
• CAD outperforms standard debiased Lasso in settings with inaccurate or correlated estimation errors, particularly in semi-supervised learning contexts.