[Paper Review] Please Stop Permuting Features: An Explanation and Alternatives.

Giles Hooker, Lucas Mentch|arXiv (Cornell University)|May 1, 2019

Explainable Artificial Intelligence (XAI)32 references87 citations

TL;DR

This paper argues that permute-and-predict (PaP) methods for interpreting black-box models are fundamentally flawed due to artificial disruption of feature dependencies, leading to misleading interpretations—especially in the presence of correlated features. It proposes alternatives like feature removal, conditional permutations, and model distillation to provide more reliable model-interpretation metrics.

ABSTRACT

This paper advocates against permute-and-predict (PaP) methods for interpreting black box functions. Methods such as the variable importance measures proposed for random forests, partial dependence plots, and individual conditional expectation plots remain popular because of their ability to provide model-agnostic measures that depend only on the pre-trained model output. However, numerous studies have found that these tools can produce diagnostics that are highly misleading, particularly when there is strong dependence among features. Rather than simply add to this growing literature by further demonstrating such issues, here we seek to provide an explanation for the observed behavior. In particular, we argue that breaking dependencies between features in hold-out data places undue emphasis on sparse regions of the feature space by forcing the original model to extrapolate to regions where there is little to no data. We explore these effects through various settings where a ground-truth is understood and find support for previous claims in the literature that PaP metrics tend to over-emphasize correlated features both in variable importance and partial dependence plots, even though applying permutation methods to the ground-truth models do not. As an alternative, we recommend more direct approaches that have proven successful in other settings: explicitly removing features, conditional permutations, or model distillation methods.

Motivation & Objective

• To explain why permute-and-predict (PaP) methods produce misleading interpretations in the presence of feature dependencies.
• To demonstrate that PaP methods overemphasize correlated features in variable importance and partial dependence plots, even when applied to ground-truth models.
• To challenge the widespread use of PaP methods despite known issues, by providing a root-cause analysis of their failure mode.
• To advocate for more reliable, direct interpretation methods such as feature removal, conditional permutations, and model distillation.
• To provide empirical evidence in controlled settings where ground-truth behavior is known, showing PaP methods fail where alternatives succeed.

Proposed method

• Analyzes the impact of feature permutation on model extrapolation by disrupting natural feature dependencies in hold-out data.
• Uses controlled experiments with known ground-truth models to isolate the effect of feature dependence on PaP diagnostics.
• Compares PaP-based metrics (e.g., variable importance, partial dependence) against alternative methods like feature removal and conditional permutations.
• Employs model distillation as a method to train surrogate models that preserve feature dependencies, enabling more faithful interpretation.
• Evaluates interpretation quality by comparing predicted feature importance and partial dependence trends against the true underlying model behavior.
• Focuses on sparse regions of feature space where PaP methods force models to extrapolate, leading to distorted outputs.

Experimental results

Research questions

• RQ1 Why do permute-and-predict methods produce misleading interpretations when features are dependent?
• RQ2 How does breaking feature dependencies during permutation lead to overestimation of correlated features in importance and partial dependence plots?
• RQ3 To what extent do PaP methods fail in controlled settings with known ground-truth models, even when applied to the true model?
• RQ4 Can alternative methods such as conditional permutations or model distillation produce more accurate and reliable model interpretations?
• RQ5 How do feature removal and model distillation compare to PaP in preserving the true functional relationships in the data?

Key findings

• Permute-and-predict methods overemphasize correlated features in variable importance and partial dependence plots due to forced extrapolation into sparse regions of the feature space.
• The same PaP methods applied to the ground-truth model still produce misleading results, indicating the flaw is inherent to the method, not model-specific error.
• Disrupting feature dependencies during permutation leads to model predictions in data-sparse regions, distorting the interpretation of feature effects.
• Conditional permutations and feature removal produce more accurate interpretations by preserving natural feature relationships and avoiding extrapolation.
• Model distillation methods that retain feature dependencies yield more faithful surrogate models, improving interpretability compared to PaP.
• In settings with known ground-truth, PaP-based diagnostics systematically deviate from true feature effects, especially under strong feature correlation.