[论文解读] Geometry- and Accuracy-Preserving Random Forest Proximities

Jake S. Rhodes, Adele Cutler+1 more|arXiv (Cornell University)|Jan 29, 2022

Anomaly Detection Techniques and Applications被引用 4

一句话总结

本文提出了随机森林-几何与准确率保持型邻近度（RF-GAP），这是一种新型的随机森林成对相似性定义，通过邻近距离加权投票精确恢复袋外预测结果。RF-GAP 确保学习到的数据几何结构与随机森林的预测结构保持一致，相较于现有方法，在插补、异常检测和可视化任务中表现更优。

ABSTRACT

Random forests are considered one of the best out-of-the-box classification and regression algorithms due to their high level of predictive performance with relatively little tuning. Pairwise proximities can be computed from a trained random forest and measure the similarity between data points relative to the supervised task. Random forest proximities have been used in many applications including the identification of variable importance, data imputation, outlier detection, and data visualization. However, existing definitions of random forest proximities do not accurately reflect the data geometry learned by the random forest. In this paper, we introduce a novel definition of random forest proximities called Random Forest-Geometry- and Accuracy-Preserving proximities (RF-GAP). We prove that the proximity-weighted sum (regression) or majority vote (classification) using RF-GAP exactly matches the out-of-bag random forest prediction, thus capturing the data geometry learned by the random forest. We empirically show that this improved geometric representation outperforms traditional random forest proximities in tasks such as data imputation and provides outlier detection and visualization results consistent with the learned data geometry.

研究动机与目标

• 解决传统随机森林邻近距离与随机森林实际学习到的数据几何结构之间的不一致性。
• 开发一种同时保持数据几何结构与随机森林模型预测准确率的邻近距离定义。
• 通过使邻近距离与袋外预测结果对齐，实现更精确且可解释的应用，如数据插补、异常检测和可视化。
• 为现有邻近距离定义提供一个理论基础扎实且经实证验证的替代方案，这些方法无法恢复森林的袋外预测结果。

提出的方法

• 使用袋内观测值作为投票点，计算袋外观测值的权重，定义 RF-GAP 邻近距离。
• 将两个袋外点之间的邻近距离定义为在每棵树中共享相同终端节点的袋内观测值的归一化计数。
• 通过邻近距离加权求和（回归）或多数票表决（分类）进行标签预测，确保与袋外预测完全一致。
• 通过与类内邻居的逆邻近距离进行异常检测，结合中位数与四分位距（MAD）实现鲁棒性调整。
• 在多维缩放（MDS）中使用邻近距离矩阵进行可视化，点的大小反映异常值得分。
• 实现一种稀疏变体以提升大规模数据集的可扩展性。

Figure 1: An example of a random forest and notation with regards to a particular observation $x_{1}$ . The red-encircled trees are those in which $x_{1}$ is out of bag, making up the set of trees $S_{1}$ . A particular tree in $S_{1}$ is exhibited. The out-of-bag indices for the tree are given in r

实验结果

研究问题

• RQ1 能否构建一种新的随机森林邻近距离定义，使得邻近距离加权预测结果与随机森林的袋外预测完全一致？
• RQ2 与现有定义相比，所提出的 RF-GAP 邻近距离度量是否更准确地反映随机森林学习到的数据几何结构？
• RQ3 RF-GAP 是否能提升下游任务（如数据插补、异常检测和可视化）的性能？
• RQ4 与传统邻近距离及 RFProxIH 相比，RF-GAP 在捕捉聚类结构以及识别误分类或异常点方面表现如何？

主要发现

• 理论上证明并实证验证了，RF-GAP 邻近距离在回归与分类任务中均能精确重现随机森林的袋外预测结果。
• 在数据插补任务中，RF-GAP 的预测准确率优于传统随机森林邻近距离与 RFProxIH。
• 基于 RF-GAP 的异常检测可生成清晰反映与类簇距离的得分，高分点对应 MNIST 数据集中模糊或误分类的数字。
• 使用 RF-GAP 进行可视化时，类簇分离更清晰，异常观测点的识别更准确，优于原始邻近距离与 RFProxIH。
• RF-GAP 的异常值得分与类内点的平均邻近距离成反比，相比现有方法，提供了更具可解释性与一致性的度量方式。
• 该方法可扩展至包含数千个样本的数据集，且提出了稀疏实现方式以进一步拓展至更大规模数据集的适用性。

Figure 2: Random forest predictions are compared here with the proximity-weighted predictions for both regression and classification problems. For each of the datasets listed in supplemental Table B.1 , the differences were calculated across five random seeds. For classification, the RF-GAP predicti