[Paper Review] Complete Analysis of a Random Forest Model

Jason M. Klusowski|arXiv (Cornell University)|May 7, 2018

Statistical Methods and Inference5 references15 citations

TL;DR

This paper provides a complete theoretical analysis of a random forest model that selects features uniformly at random and splits at the midpoint of the feature's range. It establishes a mean-squared prediction error rate of order $(n oot S-1 \ oot \ ight)^{-1/(S\log 2 + 1)}$, which improves upon the minimax rate for sparse regression when $S \leq \lfloor 0.72d \rfloor$, and shows variance decays independently of ambient dimension, even for fully grown trees.

ABSTRACT

Random forests have become an important tool for improving accuracy in regression problems since their popularization by (Breiman, 2001) and others. In this paper, we revisit a random forest model originally proposed by (Breiman, 2004) and later studied by (Biau, 2012), where a feature is selected at random and the split occurs at the midpoint of the block containing the chosen feature. If the regression function is sparse and depends only on a small, unknown subset of $ S $ out of $ d $ features, we show that given $ n $ observations, this random forest model outputs a predictor that has a mean-squared prediction error of order $ \left(n\sqrt{\log^{S-1} n} ight)^{-\frac{1}{S\log2+1}} $. When $ S \leq \lfloor 0.72 d floor $, this rate is better than the minimax optimal rate $ n^{-\frac{2}{d+2}} $ for $ d $-dimensional, Lipschitz function classes. As a consequence of our analysis, we show that the variance of the forest decays with the depth of the tree at a rate that is independent of the ambient dimension, even when the trees are fully grown. In particular, if $ \ell_{avg} $ (resp. $ \ell_{max} $) is the average (resp. maximum) number of observations per leaf node, we show that the variance of this forest is $ \Theta\left(\ell^{-1}_{avg}(\sqrt{\log n})^{-(S-1)} ight) $, which for the case of $ S = d $, is similar in form to the lower bound $ \Omega\left(\ell^{-1}_{max}(\log n)^{-(d-1)} ight) $ of (Lin and Jeon, 2006) for any random forest model with a nonadaptive splitting scheme. We also show that the bias is tight for any linear model with nonzero parameter vector. Thus, we completely characterize the fundamental limits of this random forest model. Our new analysis also implies that better theoretical performance can be achieved if the trees are grown less aggressively (i.e., grown to a shallower depth) than previous work would otherwise recommend.

Motivation & Objective

• To understand the theoretical performance limits of a specific random forest model with random feature selection and midpoint splitting.
• To analyze how the model's prediction error scales with sample size $n$ and sparsity $S$ in high-dimensional settings.
• To characterize the variance and bias behavior of the forest, especially in relation to tree depth and ambient dimension $d$.
• To determine whether less aggressive tree growth (shallower depth) could yield better theoretical performance than standard practices.

Proposed method

• The analysis focuses on a random forest model where each split selects a feature uniformly at random and splits at the midpoint of its observed range.
• The model assumes the true regression function depends only on $S$ out of $d$ features, with $S$ unknown and potentially small.
• The paper derives the mean-squared prediction error by analyzing both bias and variance components, with bias shown to be tight for linear models with nonzero coefficients.
• Variance is analyzed by deriving its decay rate as a function of average and maximum leaf size $\ell_{\text{avg}}$ and $\ell_{\text{max}}$, showing $\Theta(\ell^{-1}_{\text{avg}}(\sqrt{\log n})^{-(S-1)})$.
• The analysis compares the derived error rate to the minimax optimal rate $n^{-2/(d+2)}$ for Lipschitz functions in $d$ dimensions.
• It leverages concentration inequalities and recursive partitioning arguments to bound the bias and variance under the nonadaptive splitting scheme.

Experimental results

Research questions

• RQ1 What is the precise rate of mean-squared prediction error for this random forest model under sparse regression?
• RQ2 How does the variance of the forest decay with tree depth, and does it depend on the ambient dimension $d$?
• RQ3 Can the model achieve faster error rates than the minimax rate $n^{-2/(d+2)}$ for $d$-dimensional Lipschitz functions?
• RQ4 How does the model's performance depend on the sparsity $S$ of the true regression function?
• RQ5 Is there a theoretical advantage to growing trees less deeply than typically recommended?

Key findings

• The mean-squared prediction error of the random forest model is of order $(n\sqrt{\log^{S-1} n})^{-1/(S\log 2 + 1)}$, which is faster than the minimax rate $n^{-2/(d+2)}$ when $S \leq \lfloor 0.72d \rfloor$.
• The variance of the forest decays as $\Theta(\ell^{-1}_{\text{avg}}(\sqrt{\log n})^{-(S-1)})$, independent of the ambient dimension $d$, even for fully grown trees.
• For the case $S = d$, the variance rate matches the lower bound $\Omega(\ell^{-1}_{\text{max}}(\log n)^{-(d-1)})$ from Lin and Jeon (2006), indicating tightness.
• The bias is shown to be tight for any linear model with a nonzero parameter vector, confirming that the model can adapt to low-dimensional structure.
• The analysis reveals that growing trees less deeply than standard practice can yield better theoretical performance, challenging conventional wisdom.
• The model achieves faster convergence in sparse settings, demonstrating that random forests can adapt to intrinsic low-dimensional structure even when the ambient dimension is high.