[Paper Review] Random Forests: some methodological insights

Robin Genuer, Jean‐Michel Poggi+1 more|ArXiv.org|Nov 21, 2008

Neural Networks and ApplicationsComputer Science37 references147 citations

TL;DR

This paper provides methodological insights into Random Forests, focusing on variable importance estimation and a two-step variable selection strategy: ranking variables by importance scores followed by stepwise ascending inclusion. It confirms optimal parameter settings for standard and high-dimensional problems and demonstrates that the variable importance index reliably identifies relevant predictors, especially when combined with sequential feature introduction for improved prediction performance.

ABSTRACT

This paper examines from an experimental perspective random forests, the increasingly used statistical method for classification and regression problems introduced by Leo Breiman in 2001. It first aims at confirming, known but sparse, advice for using random forests and at proposing some complementary remarks for both standard problems as well as high dimensional ones for which the number of variables hugely exceeds the sample size. But the main contribution of this paper is twofold: to provide some insights about the behavior of the variable importance index based on random forests and in addition, to propose to investigate two classical issues of variable selection. The first one is to find important variables for interpretation and the second one is more restrictive and try to design a good prediction model. The strategy involves a ranking of explanatory variables using the random forests score of importance and a stepwise ascending variable introduction strategy.

Motivation & Objective

• To validate and extend practical guidelines for tuning Random Forest parameters in both standard (n >> p) and high-dimensional (n << p) settings.
• To investigate the behavior and reliability of the Random Forest variable importance (VI) index in identifying relevant predictors.
• To propose and evaluate a two-stage variable selection strategy: ranking variables by VI scores followed by sequential, ascending inclusion for optimal prediction.
• To address the dual goals of variable selection: interpretation (identifying important variables) and prediction (building a minimal, effective model).

Proposed method

• The study uses benchmark datasets from the R package mlbench, including real and simulated data for classification, multiclass, and regression problems.
• It evaluates Random Forest performance across various mtry values and sample sizes, focusing on prediction error and variable importance scores.
• Variable importance is computed using the out-of-bag (OOB) error reduction method, standard in Random Forests, to rank predictors.
• A stepwise ascending variable introduction strategy is applied: variables are added in order of decreasing importance until prediction performance plateaus.
• The method is tested on both standard and high-dimensional datasets, including gene expression data with p >> n.
• Statistical performance is assessed via prediction error (MSE for regression, misclassification rate for classification) and variable selection accuracy.

Experimental results

Research questions

• RQ1 How do standard Random Forest parameter settings (e.g., mtry) affect performance across different data regimes (n >> p vs. n << p)?
• RQ2 How reliable is the Random Forest variable importance index in identifying truly relevant predictors, especially in high-dimensional settings?
• RQ3 Can a two-stage variable selection strategy—ranking by importance followed by sequential inclusion—improve prediction accuracy while maintaining interpretability?
• RQ4 Does the variable importance index perform consistently across different types of data (binary, multiclass, regression) and noise levels?

Key findings

• The variable importance index in Random Forests consistently ranks the most relevant predictors first, even in high-dimensional settings with p >> n, such as gene expression data.
• For high-dimensional problems, the method achieves good prediction performance with only a small subset of top-ranked variables, confirming the utility of the importance-based selection strategy.
• The stepwise ascending inclusion of variables based on importance scores leads to stable and improved prediction performance, especially when combined with optimal mtry values.
• The study confirms that mtry values around √p or p/3 (depending on problem type) yield robust performance across diverse datasets, supporting existing heuristic recommendations.
• In multiclass and regression problems, the variable importance index successfully identifies the most predictive features, with consistent rankings across multiple runs and datasets.
• The OOB error-based variable importance measure remains reliable even when the number of predictors greatly exceeds the number of observations, supporting its use in high-dimensional feature selection.