[Paper Review] Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance

Giles Hooker, Lucas Mentch+1 more|arXiv (Cornell University)|May 1, 2019

Explainable Artificial Intelligence (XAI)Computer Science45 references38 citations

TL;DR

The paper critiques permute-and-predict methods for feature importance in black-box models, showing they can overemphasize correlated features due to extrapolation, and advocates for re-learning or conditional perturbation approaches as more reliable alternatives.

ABSTRACT

This paper reviews and advocates against the use of permute-and-predict (PaP) methods for interpreting black box functions. Methods such as the variable importance measures proposed for random forests, partial dependence plots, and individual conditional expectation plots remain popular because they are both model-agnostic and depend only on the pre-trained model output, making them computationally efficient and widely available in software. However, numerous studies have found that these tools can produce diagnostics that are highly misleading, particularly when there is strong dependence among features. The purpose of our work here is to (i) review this growing body of literature, (ii) provide further demonstrations of these drawbacks along with a detailed explanation as to why they occur, and (iii) advocate for alternative measures that involve additional modeling. In particular, we describe how breaking dependencies between features in hold-out data places undue emphasis on sparse regions of the feature space by forcing the original model to extrapolate to regions where there is little to no data. We explore these effects across various model setups and find support for previous claims in the literature that PaP metrics can vastly over-emphasize correlated features in both variable importance measures and partial dependence plots. As an alternative, we discuss and recommend more direct approaches that involve measuring the change in model performance after muting the effects of the features under investigation.

Motivation & Objective

• Review the literature showing flaws of PaP (permute-and-predict) methods for variable importance and related plots (VI, PDP, ICE).
• Demonstrate how feature dependencies cause extrapolation bias in PaP measures using simulations and explanations.
• Advocate for alternative importance measures that require additional modeling or conditional perturbation to avoid extrapolation.
• Discuss practical implications for interpreting black-box models and propose gold-standard approaches that involve re-learning or conditional perturbation.

Proposed method

• Present a simple simulated linear model with 10 features and a controlled correlation structure to illustrate bias in PaP measures.
• Train random forests and neural networks to estimate the underlying function and compute VI^π, VI^o, PDP, and ICE.
• Theoretically connect permutation importance to squared coefficients in linear models under standardization (Theorem 1).
• Visualize extrapolation effects through contour plots, ICE, and PDP analyses across varying correlation ρ and sample sizes N.
• Compare PaP with alternatives that involve conditional perturbations or re-learning (LOCO, knockoffs, model class reliance).
• Discuss the extrapolation mechanism in tree-based models and neural nets using the concept of potential nearest neighbors (pNN) and leaves.
• Provide guidance on when and how to use more robust importance measures.

Experimental results

Research questions

• RQ1 How do PaP-based variable importance measures behave when features are statistically dependent?
• RQ2 To what extent do correlated features cause PaP methods to overemphasize certain variables or misrepresent their true importance?
• RQ3 Can alternative approaches that condition on remaining features or re-learn models mitigate extrapolation bias in feature importance?
• RQ4 What theoretical connections exist between permutation importance and linear model coefficients under standardization?
• RQ5 What practical recommendations can ensure more reliable interpretation of black-box models?

Key findings

• PaP measures can heavily over-emphasize correlated features in both variable importance and partial dependence plots when features are dependent.
• In linear models with standardized features, permutation importance corresponds to the squared coefficient times the feature’s marginal sum of squares, aligning with coefficient magnitude, but this relationship breaks with flexible models (RF, NN) under correlation due to extrapolation.
• Extrapolation causes PaP to evaluate predictions in regions with little or no data, inflating the perceived importance of correlated features.
• Partial Dependence and ICE plots can misrepresent feature effects under correlation, with greater instability observed in neural networks compared to random forests.
• Alternative approaches that condition on remaining features or re-learn models (e.g., LOCO, conditional randomization tests, knockoffs) avoid extrapolation and provide more reliable importance assessments.
• A condition-and-relearn framework (conditioning + re-learning) is proposed as a gold standard for evaluating feature importance, though not entirely immune to issues.