[Paper Review] Understanding Random Forests: From Theory to Practice

Gilles Louppe|arXiv (Cornell University)|Jan 1, 2014

Neural Networks and ApplicationsComputer Science105 references594 citations

TL;DR

This PhD dissertation provides a comprehensive theoretical and practical analysis of random forests, rigorously examining their learning mechanisms, interpretability via variable importance measures, and scalability on large datasets. It identifies critical flaws in standard variable importance estimation due to masking and tree structure, proposes theoretical corrections for totally randomized trees, and demonstrates that ensemble models trained on small random subsamples can match performance on massive datasets while reducing memory usage.

ABSTRACT

Data analysis and machine learning have become an integrative part of the modern scientific methodology, offering automated procedures for the prediction of a phenomenon based on past observations, unraveling underlying patterns in data and providing insights about the problem. Yet, caution should avoid using machine learning as a black-box tool, but rather consider it as a methodology, with a rational thought process that is entirely dependent on the problem under study. In particular, the use of algorithms should ideally require a reasonable understanding of their mechanisms, properties and limitations, in order to better apprehend and interpret their results. Accordingly, the goal of this thesis is to provide an in-depth analysis of random forests, consistently calling into question each and every part of the algorithm, in order to shed new light on its learning capabilities, inner workings and interpretability. The first part of this work studies the induction of decision trees and the construction of ensembles of randomized trees, motivating their design and purpose whenever possible. Our contributions follow with an original complexity analysis of random forests, showing their good computational performance and scalability, along with an in-depth discussion of their implementation details, as contributed within Scikit-Learn. In the second part of this work, we analyse and discuss the interpretability of random forests in the eyes of variable importance measures. The core of our contributions rests in the theoretical characterization of the Mean Decrease of Impurity variable importance measure, from which we prove and derive some of its properties in the case of multiway totally randomized trees and in asymptotic conditions. In consequence of this work, our analysis demonstrates that variable importances [...].

Motivation & Objective

• To provide a rigorous theoretical and practical understanding of random forests as a machine learning methodology, moving beyond black-box usage.
• To investigate and resolve fundamental issues in variable importance estimation, particularly the bias caused by masking effects and impurity misestimation.
• To analyze the computational scalability and memory efficiency of random forests on large-scale datasets.
• To evaluate the effectiveness of simultaneous subsampling of samples and features as a practical alternative to training on full datasets.
• To contribute theoretical and empirical insights into the design and implementation of random forests, especially within the Scikit-Learn framework.

Proposed method

• Conducts a complexity analysis of random forests, evaluating computational efficiency and scalability in both theoretical and implementation contexts.
• Theoretically characterizes the Mean Decrease of Impurity (MDI) variable importance measure in the context of multiway totally randomized trees under asymptotic conditions.
• Derives mathematical properties of MDI under controlled conditions to expose inherent biases in non-totally randomized trees.
• Employs extensive empirical experiments to compare performance of models trained on full datasets versus small random subsamples of data.
• Proposes and evaluates a dual subsampling strategy—simultaneously sampling both samples and features—to reduce memory footprint without sacrificing predictive accuracy.
• Integrates theoretical findings with practical implementation details, particularly within the Scikit-Learn library, to ensure reproducibility and real-world applicability.

Experimental results

Research questions

• RQ1 What are the theoretical properties of the Mean Decrease of Impurity (MDI) variable importance measure in random forests, especially under asymptotic and totally randomized conditions?
• RQ2 Why do standard random forests exhibit biased variable importance estimates, and what are the root causes—masking effects, impurity misestimation, or binary tree structure?
• RQ3 Can random forests maintain high predictive performance when trained on small, random subsamples of large datasets, and how does this compare to full-dataset training?
• RQ4 What is the impact of simultaneously subsampling both features and samples on model performance and memory efficiency?
• RQ5 How can the interpretability of random forests be improved through theoretical corrections to variable importance measures?

Key findings

• The Mean Decrease of Impurity (MDI) variable importance measure is theoretically well-behaved and unbiased in the case of multiway totally randomized trees under asymptotic conditions.
• In standard random forests (non-totally randomized), variable importance measures suffer from significant bias due to masking effects and impurity misestimation, particularly in the presence of correlated features.
• The binary structure of decision trees contributes to the distortion of variable importance estimates, especially when features are correlated.
• Empirical results show that training random forests on small, random subsamples of data—while simultaneously subsampling features—can achieve performance comparable to full-dataset training.
• The memory requirements are significantly reduced when using subsampled data, making large-scale random forest training feasible on standard hardware.
• The study demonstrates that building ensembles from multiple models trained on small, independent subsamples is a viable and efficient alternative to training single models on massive datasets.