[Paper Review] Honest Confidence Regions for Logistic Regression with a Large Number of Controls

Alexandre Belloni, Victor Chernozhukov+1 more|arXiv (Cornell University)|Apr 15, 2013

Statistical Methods and Inference11 references20 citations

TL;DR

This paper proposes a robust method for estimating and constructing honest confidence regions for the coefficient of interest in logistic regression when the number of controls exceeds sample size. By leveraging sparsity and instrumental variable techniques, it achieves root-n estimation and uniform validity under weak regularity conditions, without relying on consistent model selection.

ABSTRACT

This paper considers generalized linear models in the presence of many controls. We lay out a general methodology to estimate an effect of interest based on the construction of an instrument that immunize against model selection mistakes and apply it to the case of logistic binary choice model. More specifically we propose new methods for estimating and constructing confidence regions for a regression parameter of primary interest $\alpha_0$, a parameter in front of the regressor of interest, such as the treatment variable or a policy variable. These methods allow to estimate $\alpha_0$ at the root-$n$ rate when the total number $p$ of other regressors, called controls, potentially exceed the sample size $n$ using sparsity assumptions. The sparsity assumption means that there is a subset of $s<n$ controls which suffices to accurately approximate the nuisance part of the regression function. Importantly, the estimators and these resulting confidence regions are valid uniformly over $s$-sparse models satisfying $s^2\log^2 p = o(n)$ and other technical conditions. These procedures do not rely on traditional consistent model selection arguments for their validity. In fact, they are robust with respect to moderate model selection mistakes in variable selection. Under suitable conditions, the estimators are semi-parametrically efficient in the sense of attaining the semi-parametric efficiency bounds for the class of models in this paper.

Motivation & Objective

• Address the challenge of estimating a parameter of interest in logistic regression when the number of control variables exceeds sample size.
• Develop inference procedures that remain valid even when model selection for controls is imperfect or inconsistent.
• Ensure estimation and confidence regions are uniformly valid across a broad class of high-dimensional, sparse models.
• Achieve semi-parametric efficiency without requiring consistent model selection or strong parametric assumptions.

Proposed method

• Construct an instrumental variable that is uncorrelated with estimation errors from model selection of controls.
• Use sparsity assumptions: only a small subset of controls (s < n) are needed to accurately approximate the nuisance regression function.
• Estimate the parameter of interest using a two-stage procedure that separates the estimation of the main effect from the nuisance function.
• Apply debiasing techniques to correct for estimation bias introduced by high-dimensional control selection.
• Construct confidence regions that are robust to moderate model selection mistakes by ensuring uniform validity over s-sparse models.
• Leverage asymptotic theory under the condition s² log²p = o(n) to ensure root-n convergence and asymptotic normality of the estimator.

Experimental results

Research questions

• RQ1 Can we construct honest confidence regions for a parameter of interest in logistic regression when the number of controls exceeds sample size?
• RQ2 How can we ensure valid inference when model selection for controls is inconsistent or imperfect?
• RQ3 What conditions allow for root-n estimation of the parameter of interest in high-dimensional logistic models?
• RQ4 Can the proposed method achieve semi-parametric efficiency without relying on consistent model selection?
• RQ5 How does the method perform uniformly across different sparse models satisfying s² log²p = o(n)?

Key findings

• The proposed estimator achieves root-n convergence for the parameter of interest even when the number of controls p exceeds sample size n.
• Confidence regions constructed using the method are uniformly valid over s-sparse models under the condition s² log²p = o(n).
• The method remains valid under moderate model selection mistakes, without requiring consistent selection of control variables.
• The estimator attains the semi-parametric efficiency bound under suitable regularity conditions, indicating optimal estimation performance.
• The approach does not rely on consistent model selection for validity, making it robust to common pitfalls in high-dimensional variable selection.
• The method applies broadly to generalized linear models with many controls, with a specific focus on logistic binary choice models.