[Paper Review] Achieving Reliable Causal Inference with Data-Mined Variables: A Random Forest Approach to the Measurement Error Problem

Mochen Yang, Edward McFowland+2 more|arXiv (Cornell University)|Dec 19, 2020

Bayesian Modeling and Causal Inference61 references4 citations

TL;DR

This paper proposes ForestIV, a novel random forest-based method to correct bias in causal inference caused by measurement error in machine learning-generated variables. By leveraging weakly correlated prediction errors across individual trees in a random forest, it uses selected trees as instrumental variables to produce consistent and reliable causal estimates, significantly outperforming existing bias-correction methods in simulations.

ABSTRACT

Combining machine learning with econometric analysis is becoming increasingly prevalent in both research and practice. A common empirical strategy involves the application of predictive modeling techniques to 'mine' variables of interest from available data, followed by the inclusion of those variables into an econometric framework, with the objective of estimating causal effects. Recent work highlights that, because the predictions from machine learning models are inevitably imperfect, econometric analyses based on the predicted variables are likely to suffer from bias due to measurement error. We propose a novel approach to mitigate these biases, leveraging the ensemble learning technique known as the random forest. We propose employing random forest not just for prediction, but also for generating instrumental variables to address the measurement error embedded in the prediction. The random forest algorithm performs best when comprised of a set of trees that are individually accurate in their predictions, yet which also make 'different' mistakes, i.e., have weakly correlated prediction errors. A key observation is that these properties are closely related to the relevance and exclusion requirements of valid instrumental variables. We design a data-driven procedure to select tuples of individual trees from a random forest, in which one tree serves as the endogenous covariate and the other trees serve as its instruments. Simulation experiments demonstrate the efficacy of the proposed approach in mitigating estimation biases and its superior performance over three alternative methods for bias correction.

Motivation & Objective

• To address the growing problem of measurement error bias in causal inference when using machine learning-predicted variables as covariates in econometric models.
• To develop a general-purpose, data-driven method that leverages the inherent structure of random forests to generate valid instrumental variables for bias correction.
• To overcome the limitations of traditional bias-correction techniques by exploiting the weakly correlated prediction errors of individual trees in a random forest ensemble.
• To provide a practical, implementable framework for researchers to achieve consistent causal estimates even when machine learning predictions are imperfect.
• To validate the method through comprehensive simulations across continuous and binary data-mined variables, demonstrating robustness and superiority over existing approaches.

Proposed method

• ForestIV uses a random forest model trained on labeled data with true outcomes to generate multiple individual tree predictions, each with distinct prediction errors.
• It identifies pairs of trees where one tree's predictions serve as the endogenous covariate and another tree's predictions serve as an instrumental variable, based on weak correlation in prediction errors.
• The method applies a two-step lasso-based selection procedure to identify the optimal set of trees that satisfy relevance and exclusion restrictions for valid instruments.
• It formulates a covariance-based criterion to assess the strength and validity of candidate instruments, ensuring low correlation between instrument errors and the error in the endogenous predictor.
• The approach is implemented through a data-driven selection algorithm that evaluates instrument quality using empirical moments of prediction errors across training samples.
• Post-estimation diagnostics include Hotelling's T² test to compare ForestIV estimates with standard labeled-data estimates and convergence plots to assess stability with increasing unlabeled data.

Figure 1: Distributions of Biased, Unbiased, and ForestIV Estimation on $lnCnt$ across 100 Simulation Runs

Experimental results

Research questions

• RQ1 Can random forest trees be used to generate valid instrumental variables that correct for measurement error in machine learning-predicted covariates?
• RQ2 How does the performance of ForestIV compare to three alternative bias-correction methods in terms of reducing coefficient bias?
• RQ3 What are the key empirical indicators that signal the convergence and validity of ForestIV estimates in finite samples?
• RQ4 To what extent does ForestIV maintain consistency and efficiency under varying levels of prediction error in the data-mined variables?
• RQ5 In what scenarios is ForestIV most effective compared to relying solely on labeled data for causal inference?

Key findings

• ForestIV significantly reduces estimation bias in both continuous and binary data-mined covariates, outperforming three benchmark bias-correction methods in simulation experiments.
• The method achieves consistent causal estimates by exploiting the weakly correlated prediction errors inherent in random forest trees, satisfying key instrumental variable conditions.
• Simulation results show that ForestIV produces coefficient estimates that are closer to the true causal effects than standard regression using predicted variables, even when prediction accuracy is moderate.
• Empirical diagnostics such as the Hotelling T² test and convergence plots provide reliable signals for instrument validity and estimation stability in finite samples.
• The approach remains effective even when the labeled data size is limited, making it suitable for real-world applications where true labels are scarce.
• ForestIV demonstrates robustness across diverse data types, including structured, text, and image-based data, due to its general-purpose design.

Figure 2: Distribution of Biased, Unbiased, and ForestIV Estimation on $Deposit$ across 100 Simulation Runs