[Paper Review] Feature Removal Is a Unifying Principle for Model Explanation Methods

Ian Covert, Scott Lundberg+1 more|arXiv (Cornell University)|Nov 6, 2020

Explainable Artificial Intelligence (XAI)57 references19 citations

TL;DR

This paper proposes a unifying framework for model explanation methods based on the principle of feature removal, showing that 26 diverse methods—including SHAP, LIME, and permutation tests—share this core mechanism. By formalizing feature removal through three dimensions—how features are removed, what model behavior is analyzed, and how influence is summarized—the framework reveals deep structural similarities across local, global, and game-theoretic approaches, enabling better method selection and guiding future research.

ABSTRACT

Researchers have proposed a wide variety of model explanation approaches, but it remains unclear how most methods are related or when one method is preferable to another. We examine the literature and find that many methods are based on a shared principle of explaining by removing - essentially, measuring the impact of removing sets of features from a model. These methods vary in several respects, so we develop a framework for removal-based explanations that characterizes each method along three dimensions: 1) how the method removes features, 2) what model behavior the method explains, and 3) how the method summarizes each feature's influence. Our framework unifies 26 existing methods, including several of the most widely used approaches (SHAP, LIME, Meaningful Perturbations, permutation tests). Exposing the fundamental similarities between these methods empowers users to reason about which tools to use, and suggests promising directions for ongoing model explainability research.

Motivation & Objective

• To clarify the relationships and underlying principles among diverse model explanation methods that have proliferated in the literature.
• To identify a unifying principle—feature removal—that underlies many existing explanation techniques.
• To develop a formal framework that characterizes explanation methods along three dimensions: feature removal strategy, model behavior analyzed, and influence summarization method.
• To systematize understanding of widely used methods (e.g., SHAP, LIME, IME) by revealing their shared foundation in feature removal.
• To empower researchers and practitioners to reason more carefully about method selection and to guide future research in model explainability.

Proposed method

• The framework models explanation methods as operating through three core choices: (1) how features are removed (e.g., replacing with baseline, marginalizing, or training separate models), (2) what model behavior is analyzed (e.g., prediction, loss, or variance), and (3) how feature influence is summarized (e.g., via expectation, difference, or Shapley values).
• Subset functions are introduced as a mathematical tool to represent and compare different feature removal strategies, enabling interchangeability across methods.
• The framework generalizes existing methods by disentangling their core aims from practical approximations, such as using marginal or conditional distributions for feature removal.
• It formalizes the connection between global and local methods, showing that even methods like Shapley Effects and SAGE can be expressed within the same feature removal paradigm.
• The framework is applied to 26 methods, including SHAP, LIME, Meaningful Perturbations, permutation tests, and TreeSHAP, demonstrating their shared foundation.
• The paper uses mathematical derivations to show that methods like KernelSHAP and IME are equivalent under specific assumptions, such as using the joint marginal distribution for removed features.

Experimental results

Research questions

• RQ1 How are widely used model explanation methods such as SHAP, LIME, and permutation tests related at a fundamental level?
• RQ2 Can a unifying framework be developed that captures the core mechanism shared across diverse explanation methods?
• RQ3 What are the three key dimensions that differentiate existing explanation methods while preserving a common principle?
• RQ4 To what extent do methods that appear unrelated—such as game-theoretic approaches and feature selection techniques—share the same underlying mechanism?
• RQ5 How can the framework help clarify the assumptions and trade-offs of existing methods, such as the choice of distribution for marginalizing removed features?

Key findings

• The paper identifies feature removal as a unifying principle across 26 existing explanation methods, including SHAP, LIME, and permutation tests, revealing that they all operate by measuring the impact of removing features from a model.
• The framework formalizes that all methods can be characterized by three choices: feature removal strategy, model behavior analyzed, and influence summarization, enabling systematic comparison.
• The study shows that methods like IME and SHAP are mathematically equivalent under certain assumptions, such as using the joint marginal distribution for removed features.
• The framework reveals that TreeSHAP’s handling of missing features is not equivalent to standard marginalization but instead uses a path-averaging mechanism based on tree structure.
• The paper demonstrates that variance-based methods like Shapley Effects and SAGE can be recast within the feature removal framework by redefining the cooperative game in terms of loss or variance reduction.
• By disentangling the core aims of methods from their computational approximations, the framework exposes that some widely used methods rely on implicit assumptions—such as uniform or marginal distributions—that are not always clearly stated in original works.