[Paper Review] Splitting matters: how monotone transformation of predictor variables may improve the predictions of decision tree models

Tal Galili, Isaac Meilijson|arXiv (Cornell University)|Nov 14, 2016

Machine Learning and Data Classification3 citations

TL;DR

This paper challenges the common belief that decision tree models are invariant to monotone transformations of predictors, showing that interpolation errors near split points can be reduced by using mid-point interpolation or quantile transformation. It demonstrates that transforming predictors via their cumulative distribution function (CDF) can cut interpolation error by up to 50%, especially when training and test data have different distributions.

ABSTRACT

It is widely believed that the prediction accuracy of decision tree models is invariant under any strictly monotone transformation of the individual predictor variables. However, this statement may be false when predicting new observations with values that were not seen in the training-set and are close to the location of the split point of a tree rule. The sensitivity of the prediction error to the split point interpolation is high when the split point of the tree is estimated based on very few observations, reaching 9% misclassification error when only 10 observations are used for constructing a split, and shrinking to 1% when relying on 100 observations. This study compares the performance of alternative methods for split point interpolation and concludes that the best choice is taking the mid-point between the two closest points to the split point of the tree. Furthermore, if the (continuous) distribution of the predictor variable is known, then using its probability integral for transforming the variable ("quantile transformation") will reduce the model's interpolation error by up to about a half on average. Accordingly, this study provides guidelines for both developers and users of decision tree models (including bagging and random forest).

Motivation & Objective

• To investigate whether decision tree models are truly invariant under monotone transformations of predictor variables, especially in out-of-sample interpolation.
• To analyze the impact of split point interpolation methods on prediction error when new observations are near but not in the training data.
• To evaluate the effectiveness of different interpolation rules—Sweep Left, Sweep Right, and mid-point—under varying training data conditions.
• To explore whether transforming predictors using their empirical or theoretical CDF (probability integral transformation) reduces interpolation error.
• To provide practical guidelines for improving decision tree, random forest, and bagging models through optimal predictor transformation and interpolation

Proposed method

• Proposes three interpolation rules: Sweep Left (classifies above the smallest value), Sweep Right (classifies only at the largest value), and mid-point interpolation (uses the average of the two closest values to the split point).
• Uses analytical and simulation-based evaluation of mean absolute error (MAE) across different distributions (uniform, beta, bi-triangular, mixture of normals) to compare interpolation methods.
• Applies the probability integral transform (i.e., $F(X)$) to convert predictor variables into uniform variates, assuming known or estimated CDFs, to stabilize interpolation performance.
• Employs the empirical CDF (ECDF) for semi-parametric correction in cases where the true distribution is unknown but unlabeled test data is available.
• Uses simulation studies with $10^5$ iterations to estimate MAE for various estimators, including Rao-Blackwellized and swept estimators.
• Validates findings on real-world data (e.g., weatherAUS dataset) using R code and the `edfun` package for fast empirical CDF computation

Experimental results

Research questions

• RQ1 Does the prediction accuracy of decision trees depend on the choice of split point interpolation method when new observations are near but not in the training data?
• RQ2 How does the misclassification error vary with the number of training observations used to estimate a split point, particularly near the split boundary?
• RQ3 Can transforming predictor variables using their cumulative distribution function (CDF) reduce interpolation error in decision tree models?
• RQ4 What is the performance gain of using the mid-point interpolation rule compared to Sweep Left or Sweep Right when training and test data have different distributions?
• RQ5 To what extent can the probability integral transformation improve prediction accuracy in semi-supervised or out-of-distribution settings?

Key findings

• Prediction error near split points can be as high as 9% when only 10 training observations are used to estimate a split, dropping to 1% with 100 observations.
• The mid-point interpolation rule ($\hat{p}_{SB} = \frac{L+R}{2}$) consistently outperforms both Sweep Left and Sweep Right, reducing MAE by up to 50% in some cases.
• When training and test data come from different distributions (e.g., $X_{train} \sim BT(p)$, $X_{test} \sim U(0,1)$), using the probability integral transform ($F_{train}(X_{train})$) reduces interpolation error by up to half on average.
• The performance of the mid-point rule is robust and often better than the theoretical Bayes estimator, especially when the true distribution is unknown.
• Using the empirical CDF (ECDF) for transformation in semi-parametric settings can significantly reduce MAE, particularly when unlabeled test data is available for distribution estimation.
• The study confirms that the widely held belief of invariance under monotone transformations is invalid in interpolation scenarios, especially when training data is sparse or mismatched with test data