[Paper Review] Geometry- and Accuracy-Preserving Random Forest Proximities

Jake S. Rhodes, Adele Cutler+1 more|arXiv (Cornell University)|Jan 29, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper introduces Random Forest-Geometry- and Accuracy-Preserving proximities (RF-GAP), a novel definition of random forest pairwise similarities that exactly recovers out-of-bag predictions through proximity-weighted voting. RF-GAP ensures the learned data geometry aligns with the random forest’s predictive structure, improving performance in imputation, outlier detection, and visualization compared to existing methods.

ABSTRACT

Random forests are considered one of the best out-of-the-box classification and regression algorithms due to their high level of predictive performance with relatively little tuning. Pairwise proximities can be computed from a trained random forest and measure the similarity between data points relative to the supervised task. Random forest proximities have been used in many applications including the identification of variable importance, data imputation, outlier detection, and data visualization. However, existing definitions of random forest proximities do not accurately reflect the data geometry learned by the random forest. In this paper, we introduce a novel definition of random forest proximities called Random Forest-Geometry- and Accuracy-Preserving proximities (RF-GAP). We prove that the proximity-weighted sum (regression) or majority vote (classification) using RF-GAP exactly matches the out-of-bag random forest prediction, thus capturing the data geometry learned by the random forest. We empirically show that this improved geometric representation outperforms traditional random forest proximities in tasks such as data imputation and provides outlier detection and visualization results consistent with the learned data geometry.

Motivation & Objective

• To address the inconsistency between traditional random forest proximities and the actual data geometry learned by the random forest.
• To develop a proximity definition that preserves both the geometric structure and predictive accuracy of the random forest model.
• To enable more accurate and interpretable applications such as data imputation, outlier detection, and visualization by aligning proximities with out-of-bag predictions.
• To provide a theoretically grounded and empirically validated alternative to existing proximity definitions that fail to recover the forest’s out-of-bag predictions.

Proposed method

• Define RF-GAP proximities using in-bag observations as voting points to compute weights for out-of-bag observations.
• Compute proximity between two out-of-bag points as the normalized count of in-bag observations that share the same terminal node in each tree.
• Use proximity-weighted sums (regression) or majority votes (classification) to predict labels, ensuring exact match with out-of-bag predictions.
• Apply proximity-based outlier detection via inverse proximity to within-class neighbors, scaled by median and MAD for robustness.
• Use proximity matrices in multidimensional scaling (MDS) for visualization, with point sizes reflecting outlier scores.
• Implement a sparse variant to improve scalability for larger datasets.

Figure 1: An example of a random forest and notation with regards to a particular observation $x_{1}$ . The red-encircled trees are those in which $x_{1}$ is out of bag, making up the set of trees $S_{1}$ . A particular tree in $S_{1}$ is exhibited. The out-of-bag indices for the tree are given in r

Experimental results

Research questions

• RQ1 Can a new definition of random forest proximities be constructed such that proximity-weighted predictions exactly match the out-of-bag predictions of the random forest?
• RQ2 Does the proposed RF-GAP proximity measure better reflect the data geometry learned by the random forest compared to existing definitions?
• RQ3 Can RF-GAP improve performance in downstream tasks such as data imputation, outlier detection, and visualization?
• RQ4 How does RF-GAP compare to traditional proximities and RFProxIH in capturing cluster structure and identifying misclassified or outlying points?

Key findings

• RF-GAP proximities exactly reproduce the out-of-bag predictions of the random forest in both regression and classification tasks, as proven theoretically and verified empirically.
• In data imputation tasks, RF-GAP outperforms traditional random forest proximities and RFProxIH, demonstrating improved predictive accuracy.
• Outlier detection using RF-GAP produces scores that clearly reflect distance from class clusters, with high-scoring points corresponding to ambiguous or misclassified digits in MNIST.
• Visualization using RF-GAP shows clearer separation of class clusters and more accurate identification of outlying observations compared to original proximities and RFProxIH.
• The outlier score in RF-GAP is inversely proportional to average proximity to within-class points, providing a more interpretable and consistent measure than existing methods.
• The method is scalable to datasets with a few thousand observations, and a sparse implementation is proposed to extend applicability to larger datasets.

Figure 2: Random forest predictions are compared here with the proximity-weighted predictions for both regression and classification problems. For each of the datasets listed in supplemental Table B.1 , the differences were calculated across five random seeds. For classification, the RF-GAP predicti