[Paper Review] Guided Random Forest in the RRF Package

Houtao Deng|arXiv (Cornell University)|Jun 2, 2013

Gene expression and cancer classification6 references61 citations

TL;DR

This paper proposes Guided Random Forest (GRF), a parallelizable feature selection method that uses importance scores from a standard Random Forest to guide tree construction, significantly reducing feature count while improving classification accuracy. On 10 high-dimensional gene data sets, GRF-RF (Random Forest on GRF-selected features) outperformed full RF on 9 datasets, with 7 showing statistically significant improvements at the 0.05 level.

ABSTRACT

Random Forest (RF) is a powerful supervised learner and has been popularly used in many applications such as bioinformatics. In this work we propose the guided random forest (GRF) for feature selection. Similar to a feature selection method called guided regularized random forest (GRRF), GRF is built using the importance scores from an ordinary RF. However, the trees in GRRF are built sequentially, are highly correlated and do not allow for parallel computing, while the trees in GRF are built independently and can be implemented in parallel. Experiments on 10 high-dimensional gene data sets show that, with a fixed parameter value (without tuning the parameter), RF applied to features selected by GRF outperforms RF applied to all features on 9 data sets and 7 of them have significant differences at the 0.05 level. Therefore, both accuracy and interpretability are significantly improved. GRF selects more features than GRRF, however, leads to better classification accuracy. Note in this work the guided random forest is guided by the importance scores from an ordinary random forest, however, it can also be guided by other methods such as human insights (by specifying $λ_i$). GRF can be used in "RRF" v1.4 (and later versions), a package that also includes the regularized random forest methods.

Motivation & Objective

• To address the challenge of high-dimensional feature spaces in gene expression data, where standard Random Forests struggle with interpretability and computational efficiency.
• To develop a feature selection method that leverages feature importance scores from a standard Random Forest to guide tree construction, improving model accuracy and sparsity.
• To overcome the limitations of sequential tree building in Guided Regularized Random Forest (GRRF), which restricts parallelization and increases correlation between trees.
• To enable parallel computation of trees while maintaining feature relevance and reducing redundancy through importance-weighted splitting criteria.

Proposed method

• GRF modifies the Gini importance gain at each node by weighting it with a normalized importance score from a standard Random Forest: $ gain_G(X_i) = \lambda_i \cdot gain(X_i) $.
• The weighting factor $ \lambda_i $ is defined as $ \lambda_i = 1 - \gamma + \gamma \cdot \frac{Imp_i}{Imp^*} $, where $ \gamma \in [0,1] $ controls the influence of importance scores.
• In this work, $ \gamma = 1 $ is used, so $ \lambda_i = \frac{Imp_i}{Imp^*} $, effectively penalizing low-importance features more heavily.
• Trees in GRF are built independently, enabling full parallelization, unlike GRRF, which builds trees sequentially and suffers from high correlation.
• Feature selection is performed by identifying the most frequently used features across all trees in the GRF ensemble.
• The final model applies standard Random Forest on the subset of features selected by GRF, referred to as GRF-RF.

Experimental results

Research questions

• RQ1 Can a feature selection method based on Random Forest importance scores improve classification accuracy while reducing feature dimensionality?
• RQ2 Does a parallelizable feature selection approach like GRF outperform sequential methods such as GRRF in terms of accuracy and computational efficiency?
• RQ3 Is GRF-RF (Random Forest on GRF-selected features) more accurate than standard Random Forest when applied to high-dimensional gene expression data?
• RQ4 How does the use of normalized importance scores as a penalty term affect feature selection and model performance?

Key findings

• GRF-RF significantly outperformed standard Random Forest on 9 out of 10 high-dimensional gene data sets, with 7 of these differences being statistically significant at the 0.05 level.
• On average, GRF selected only 196 features out of 500 in a simulated data set, while standard RF used all 500, and GRF-RF achieved 34 misclassifications compared to 54 for full RF.
• GRF used fewer features than GRRF in all data sets, and GRF-RF consistently outperformed both GRRF and GRRF-RF in terms of error rate across all 10 data sets.
• GRF-RF achieved lower error rates than GRF itself, indicating that applying standard Random Forest to GRF-selected features yields better performance than using GRF as a standalone classifier.
• The number of features selected by GRF was substantially lower than that of standard RF, with median feature usage reduced to less than half in most data sets, enhancing model interpretability.
• The method is robust to parameter choice, as using $ \gamma = 1 $ (maximum penalty on low-importance features) yielded strong results without tuning, suggesting practical utility in real-world applications.