[论文解读] MDA for random forests: inconsistency, and a practical solution via the Sobol-MDA

Clément Bénard, Sébastien da Veiga+1 more|arXiv (Cornell University)|Feb 26, 2021

Probabilistic and Robust Engineering Design参考文献 49被引用 6

一句话总结

本文揭示了随机森林中均值减少准确率（MDA）变量重要性度量的根本不一致性，表明在协变量相关时其收敛至有偏估计。为解决此问题，作者提出Sobol-MDA——一种计算高效、一致的替代方法，通过与Sobol敏感性指数对齐，正确估计变量重要性，在模拟和真实数据的变量选择任务中优于现有方法。

ABSTRACT

Variable importance measures are the main tools to analyze the black-box mechanisms of random forests. Although the mean decrease accuracy (MDA) is widely accepted as the most efficient variable importance measure for random forests, little is known about its statistical properties. In fact, the definition of MDA varies across the main random forest software. In this article, our objective is to rigorously analyze the behavior of the main MDA implementations. Consequently, we mathematically formalize the various implemented MDA algorithms, and then establish their limits when the sample size increases. This asymptotic analysis reveals that these MDA versions differ as importance measures, since they converge towards different quantities. More importantly, we break down these limits into three components: the first two terms are related to Sobol indices, which are well-defined measures of a covariate contribution to the response variance, widely used in the sensitivity analysis field, as opposed to the third term, whose value increases with dependence within covariates. Thus, we theoretically demonstrate that the MDA does not target the right quantity to detect influential covariates in a dependent setting, a fact that has already been noticed experimentally. To address this issue, we define a new importance measure for random forests, the Sobol-MDA, which fixes the flaws of the original MDA, and consistently estimates the accuracy decrease of the forest retrained without a given covariate, but with an efficient computational cost. The Sobol-MDA empirically outperforms its competitors on both simulated and real data for variable selection. An open source implementation in R and C++ is available online.

研究动机与目标

• 严格分析随机森林中现有MDA实现的渐近行为。
• 识别当协变量存在依赖时MDA为何产生不一致的变量重要性排序。
• 开发一种理论坚实且计算高效的MDA替代方法，确保在依赖条件下正确估计变量重要性。
• 通过实证验证新度量在变量选择任务中的优越性。

提出的方法

• 数学形式化MDA的多种实现方式，并推导其渐近极限。
• 将极限MDA值分解为三个部分：两个与Sobol指数相关，一个由协变量依赖驱动。
• 提出Sobol-MDA作为新重要性度量，仅保留基于Sobol指数的分量，确保一致性。
• 使用Sobol指数的蒙特卡洛估计高效计算变量重要性，无需重新训练随机森林。
• 证明Sobol-MDA具有一致性，且收敛于排除某一协变量时的真实准确率下降值。
• 基于ranger随机森林框架，使用R和C++实现该方法，以支持实际应用。

实验结果

研究问题

• RQ1 在协变量依赖存在时，MDA的不同实现方式在渐近行为上如何表现？
• RQ2 为何当协变量存在依赖时MDA无法正确对影响变量进行排序？
• RQ3 能否构建一种新的变量重要性度量，使其在依赖条件下一致估计变量对模型准确率的真实影响？
• RQ4 所提出的Sobol-MDA在真实和模拟数据的变量选择任务中是否优于现有MDA变体？
• RQ5 Sobol-MDA的计算成本是否与标准MDA相当，同时保持理论一致性？

主要发现

• 标准MDA收敛至一个结合Sobol指数与依赖诱导偏差项的极限值，导致在协变量依赖下不一致。
• 当协变量间相关性超过√2/2 ≈ 0.707时，依赖诱导项占主导地位，导致错误的变量排序。
• Sobol-MDA被证明具有一致性，且收敛于排除某一变量时的真实准确率下降值。
• 在模拟数据中，Sobol-MDA即使在高相关性下仍能正确识别相关变量，而标准MDA则失败。
• 在真实世界数据集中，Sobol-MDA在变量选择准确率上优于MDA及其他竞争方法。
• Sobol-MDA在R和C++中的开源实现支持高效部署，计算开销极低。