[Paper Review] Universal Consistency of Decision Trees in High Dimensions

Jason M. Klusowski|arXiv (Cornell University)|Apr 28, 2021

Statistical Methods and Inference27 references4 citations

TL;DR

This paper establishes universal consistency of CART decision trees and random forests in high-dimensional additive models, even when the number of predictors grows exponentially with sample size, under $1$-norm sparsity constraints. Remarkably, consistency is achieved with a single tree and holds even when the mtry parameter vanishes, demonstrating strong adaptivity to sparsity without ensemble averaging.

ABSTRACT

This paper shows that decision trees constructed with Classification and Regression Trees (CART) methodology are universally consistent in an additive model context, even when the number of predictor variables scales exponentially with the sample size, under certain $1$-norm sparsity constraints. The consistency is universal in the sense that there are no a priori assumptions on the distribution of the predictor variables. Amazingly, this adaptivity to (approximate or exact) sparsity is achieved with a single tree, as opposed to what might be expected for an ensemble. Finally, we show that these qualitative properties of individual trees are inherited by Breiman's random forests. Another surprise is that consistency holds even when the mtry tuning parameter vanishes as a fraction of the number of predictor variables, thus speeding up computation of the forest. A key step in the analysis is the establishment of an oracle inequality, which precisely characterizes the goodness-of-fit and complexity tradeoff for a misspecified model.

Motivation & Objective

• To establish universal consistency of decision trees under minimal assumptions on predictor distributions.
• To analyze the performance of CART trees in high-dimensional settings where the number of predictors scales exponentially with sample size.
• To investigate whether individual trees can adapt to sparsity without requiring ensemble methods.
• To extend consistency results to Breiman's random forests, including under vanishing mtry settings.
• To derive an oracle inequality that characterizes the tradeoff between model fit and complexity in misspecified models.

Proposed method

• The analysis relies on an oracle inequality that quantifies the tradeoff between goodness-of-fit and model complexity in misspecified additive models.
• The authors impose $1$-norm sparsity constraints on the true underlying regression function to control model complexity.
• Consistency is established under minimal distributional assumptions on the predictors, ensuring universal applicability.
• The method leverages structural properties of decision trees to adapt to sparse or approximately sparse additive models.
• The framework is extended to random forests by analyzing the behavior of individual trees within the ensemble.
• The mtry parameter is allowed to vanish as a fraction of the number of predictors, enabling faster computation while preserving consistency.

Experimental results

Research questions

• RQ1 Can CART decision trees achieve universal consistency in high-dimensional additive models when the number of predictors grows exponentially with sample size?
• RQ2 Does a single decision tree adapt to sparsity without requiring ensemble averaging, even under weak distributional assumptions on predictors?
• RQ3 Is universal consistency preserved in random forests when the mtry parameter vanishes relative to the number of predictors?
• RQ4 How does the tradeoff between model fit and complexity manifest in misspecified additive models?
• RQ5 What is the role of the oracle inequality in characterizing the performance of decision trees under model misspecification?

Key findings

• CART decision trees are universally consistent in high-dimensional additive models, even when the number of predictors grows exponentially with sample size.
• The consistency holds without any a priori assumptions on the distribution of the predictor variables, establishing universality.
• Adaptivity to sparsity is achieved with a single tree, contrary to expectations that ensembles are required for such adaptation.
• Random forests inherit the consistency properties of individual trees, even when the mtry parameter vanishes as a fraction of the number of predictors.
• An oracle inequality is derived that precisely characterizes the tradeoff between model fit and complexity in misspecified models.
• The results demonstrate that computational efficiency (via vanishing mtry) does not compromise statistical consistency.