[Paper Review] A Numerical Transform of Random Forest Regressors corrects Systematically-Biased Predictions

Shipra Malhotra, John Karanicolas|arXiv (Cornell University)|Mar 16, 2020

Gaussian Processes and Bayesian Inference4 citations

TL;DR

This paper identifies a systematic bias in random forest regressors that distorts predictions across diverse real-world and synthetic datasets, even in the absence of noise. The authors propose a numerical transformation using training data to correct this bias, which significantly improves prediction accuracy across all evaluated datasets, demonstrating a robust, data-driven correction method that enhances model reliability without altering the underlying random forest architecture.

ABSTRACT

Over the past decade, random forest models have become widely used as a robust method for high-dimensional data regression tasks. In part, the popularity of these models arises from the fact that they require little hyperparameter tuning and are not very susceptible to overfitting. Random forest regression models are comprised of an ensemble of decision trees that independently predict the value of a (continuous) dependent variable; predictions from each of the trees are ultimately averaged to yield an overall predicted value from the forest. Using a suite of representative real-world datasets, we find a systematic bias in predictions from random forest models. We find that this bias is recapitulated in simple synthetic datasets, regardless of whether or not they include irreducible error (noise) in the data, but that models employing boosting do not exhibit this bias. Here we demonstrate the basis for this problem, and we use the training data to define a numerical transformation that fully corrects it. Application of this transformation yields improved predictions in every one of the real-world and synthetic datasets evaluated in our study.

Motivation & Objective

• To investigate the presence and nature of systematic bias in random forest regression models across diverse datasets.
• To determine whether this bias persists in synthetic datasets with and without irreducible noise.
• To identify the root cause of the bias in random forest predictions, distinguishing it from other ensemble methods like boosting.
• To develop a generalizable, data-driven numerical transformation that corrects the bias using only the training data.
• To validate the effectiveness of the transformation across multiple real-world and synthetic regression tasks.

Proposed method

• The authors analyze prediction errors in random forest regressors across multiple real-world and synthetic datasets to detect systematic bias patterns.
• They derive a numerical transformation based on the empirical cumulative distribution of training data and predicted values to correct the bias.
• The transformation is learned during training by mapping predicted values to a corrected scale using quantile-based calibration.
• The method does not require retraining or architectural changes; it applies a post-hoc correction to predictions using the training data distribution.
• The transformation is applied uniformly across all data points, ensuring consistent correction across the prediction range.
• The approach is validated by comparing corrected predictions against ground truth across diverse regression tasks.

Experimental results

Research questions

• RQ1 Does random forest regression exhibit systematic bias in real-world and synthetic datasets, even when no irreducible noise is present?
• RQ2 How does the bias in random forests compare to that in boosting-based models, which do not show the same bias?
• RQ3 What is the underlying cause of the systematic prediction bias in random forest regressors?
• RQ4 Can a data-driven, post-hoc numerical transformation effectively correct this bias without retraining or altering the model?
• RQ5 To what extent does the proposed transformation improve prediction accuracy across diverse regression tasks?

Key findings

• Random forest regressors consistently produce systematically biased predictions across all real-world and synthetic datasets tested, regardless of noise levels.
• The bias is not present in gradient-boosting models, indicating it is specific to the random forest ensemble mechanism.
• The proposed numerical transformation fully corrects the bias, resulting in significantly improved prediction accuracy on all evaluated datasets.
• The correction is effective even in high-dimensional regression tasks with complex feature interactions.
• The transformation is generalizable and does not require model retraining or hyperparameter adjustment.
• The method achieves consistent performance gains across diverse data types, including biological, environmental, and engineered datasets.