[Paper Review] Optimal Weighted Random Forests

Xinyu Chen, Dalei Yu+1 more|arXiv (Cornell University)|May 17, 2023

Statistical Methods and Inference4 citations

TL;DR

This paper proposes two optimal weighting strategies—1 Step and 2 Steps Optimal Weighted Random Forests—for regression tasks, using Mallows-type criteria to assign adaptive weights to individual trees based on their predictive performance. The methods achieve asymptotic optimality by minimizing squared error risk and outperform equal-weighted and existing weighted RFs on real-world datasets.

ABSTRACT

The random forest (RF) algorithm has become a very popular prediction method for its great flexibility and promising accuracy. In RF, it is conventional to put equal weights on all the base learners (trees) to aggregate their predictions. However, the predictive performances of different trees within the forest can be very different due to the randomization of the embedded bootstrap sampling and feature selection. In this paper, we focus on RF for regression and propose two optimal weighting algorithms, namely the 1 Step Optimal Weighted RF (1step-WRF$_\mathrm{opt}$) and 2 Steps Optimal Weighted RF (2steps-WRF$_\mathrm{opt}$), that combine the base learners through the weights determined by weight choice criteria. Under some regularity conditions, we show that these algorithms are asymptotically optimal in the sense that the resulting squared loss and risk are asymptotically identical to those of the infeasible but best possible model averaging estimator. Numerical studies conducted on real-world data sets indicate that these algorithms outperform the equal-weight forest and two other weighted RFs proposed in existing literature in most cases.

Motivation & Objective

• To address the suboptimal performance of equal-weighted random forests by assigning adaptive weights to individual trees based on their predictive strength.
• To develop theoretically grounded weighting strategies that achieve asymptotic optimality in minimizing squared error risk for regression.
• To reduce computational cost while maintaining optimal performance through a two-step optimization framework.
• To extend model averaging principles from general machine learning to random forests, particularly for regression.
• To provide a theoretically justified, computationally efficient alternative to existing weighted RF methods.

Proposed method

• Proposes 1step-WRF opt and 2steps-WRF opt, two algorithms that assign weights to base learners (trees) using Mallows-type criteria to minimize prediction error.
• Uses a quadratic optimization framework to determine optimal weights, ensuring the resulting estimator is asymptotically equivalent to the infeasible best possible model averaging estimator.
• Introduces a two-step algorithm that reduces computational burden by limiting optimization to two quadratic tasks, improving scalability.
• Employs conditional expectations and regularity conditions (C.1–C.4) to establish theoretical properties under heteroscedastic error structures.
• Applies the Law of Iterated Expectation, Pull-out Rule, and Lebesgue Dominated Convergence Theorem to prove asymptotic optimality.
• Derives theoretical bounds on the risk ratio between the weighted estimator and the oracle model, showing convergence in probability.

Experimental results

Research questions

• RQ1 Can adaptive weighting of individual trees in random forests lead to improved regression performance compared to equal weighting?
• RQ2 Is it possible to construct a theoretically optimal weighting scheme for random forests that asymptotically matches the performance of the best possible model averaging estimator?
• RQ3 How can computational complexity be reduced while maintaining theoretical optimality in weighted random forest estimation?
• RQ4 Do the proposed weighting strategies outperform existing weighted RF methods in terms of prediction accuracy on real-world regression datasets?
• RQ5 What theoretical conditions ensure the asymptotic optimality of the weighted random forest estimator under heteroscedastic errors?

Key findings

• The proposed 1step-WRF opt and 2steps-WRF opt methods achieve asymptotic optimality, meaning their risk converges to that of the infeasible best possible model averaging estimator.
• The 2steps-WRF opt algorithm reduces computational cost by requiring only two quadratic optimization tasks, making it scalable for large forests.
• Numerical experiments on 12 UCI datasets show that both proposed methods outperform equal-weighted random forests and two existing weighted RF methods in most cases.
• Theoretical analysis confirms that the risk of the weighted estimator is bounded and converges to the optimal risk under regularity conditions, including heteroscedasticity.
• The methods are robust to model misspecification and maintain strong performance even when individual tree predictions vary significantly in quality.
• The use of Mallows-type criteria for weight selection leads to a significant reduction in mean squared error compared to baseline approaches.