[Paper Review] Debiased Machine Learning without Sample-Splitting for Stable Estimators

Qizhao Chen, Vasilis Syrgkanis+1 more|arXiv (Cornell University)|Jun 3, 2022

Statistical Methods and Inference4 citations

TL;DR

This paper proposes a debiased machine learning framework that achieves root-$n$ consistency and asymptotic normality for causal estimators without requiring sample splitting, provided the first-stage estimation algorithm satisfies an $o(n^{-1/2})$ leave-one-out stability condition. The key contribution is enabling sample reuse in moderate-sized samples, with theoretical guarantees extending to ensemble methods like bagging with sub-sampling without replacement.

ABSTRACT

Estimation and inference on causal parameters is typically reduced to a generalized method of moments problem, which involves auxiliary functions that correspond to solutions to a regression or classification problem. Recent line of work on debiased machine learning shows how one can use generic machine learning estimators for these auxiliary problems, while maintaining asymptotic normality and root-$n$ consistency of the target parameter of interest, while only requiring mean-squared-error guarantees from the auxiliary estimation algorithms. The literature typically requires that these auxiliary problems are fitted on a separate sample or in a cross-fitting manner. We show that when these auxiliary estimation algorithms satisfy natural leave-one-out stability properties, then sample splitting is not required. This allows for sample re-use, which can be beneficial in moderately sized sample regimes. For instance, we show that the stability properties that we propose are satisfied for ensemble bagged estimators, built via sub-sampling without replacement, a popular technique in machine learning practice.

Motivation & Objective

• To eliminate the need for sample splitting in debiased machine learning while preserving asymptotic normality and root-$n$ consistency of causal estimators.
• To identify minimal regularity conditions—specifically, leave-one-out stability—under which standard two-stage estimation without sample splitting remains valid.
• To extend theoretical guarantees to practical machine learning methods such as bagged estimators formed via sub-sampling without replacement.
• To provide a theoretical foundation for using complex, non-parametric machine learning models in causal inference without sacrificing inferential validity.

Proposed method

• Introduces an $o(n^{-1/2})$ leave-one-out stability condition on the first-stage nuisance estimator $\hat{g}$, which replaces the traditional Donsker or mean-squared-error-only conditions.
• Establishes that if the first-stage estimator is stable in the leave-one-out sense, then the influence of estimation error on the target parameter $\theta$ is second-order, enabling asymptotic normality.
• Uses a main lemma to show that the empirical moment process $G_n(\hat{\theta}, \hat{g})$ can be approximated by $G_n(\theta_0, g_0)$ up to $o_p(n^{-1/2})$ under stability and continuity conditions.
• Applies the framework to nonlinear moment functions by extending the stability and continuity conditions to the Jacobian and moment functions.
• Demonstrates that bagging estimators formed via sub-sampling without replacement satisfy the required stability condition when the sub-sample size $m = o(\sqrt{n})$.
• Employs stochastic equicontinuity and empirical process theory to control the deviation of the empirical moment from its expectation under the true parameter.

Experimental results

Research questions

• RQ1 Can asymptotic normality and root-$n$ consistency be achieved in debiased machine learning without sample splitting?
• RQ2 What minimal regularity conditions on the first-stage estimator are sufficient to ensure valid inference in two-stage estimation?
• RQ3 Do ensemble methods like bagging with sub-sampling without replacement satisfy the required stability conditions for inference without sample splitting?
• RQ4 How does the stability of the nuisance estimator affect the asymptotic distribution of the target parameter in moment condition estimation?

Key findings

• The paper establishes that $o(n^{-1/2})$ leave-one-out stability of the nuisance estimator $\hat{g}$ is sufficient for asymptotic normality and root-$n$ consistency of $\hat{\theta}$ without sample splitting.
• Bagging estimators formed via sub-sampling without replacement satisfy the required stability condition when the sub-sample size $m$ grows slower than $\sqrt{n}$.
• The method allows full sample reuse, improving efficiency in moderate-sized samples where sample splitting would lead to significant data loss.
• The theoretical framework extends to nonlinear moment functions under Lipschitz-type continuity conditions on the moment and Jacobian functions with respect to the nuisance function.
• The stability condition is weaker than prior work, which required faster bias decay (e.g., $o(n^{-1})$), making it applicable to a broader class of machine learning models.