[Paper Review] The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets

Oana-Maria Camburu, Eleonora Giunchiglia+3 more|arXiv (Cornell University)|Sep 23, 2020

Explainable Artificial Intelligence (XAI)11 citations

TL;DR

This paper demonstrates that feature-based explanations for model predictions are inherently ambiguous, as multiple ground-truth explanations can exist even for simple models. It reveals that Shapley values and minimal sufficient subsets (MSS) explainers target fundamentally different types of explanations—Shapley values measure feature contribution via additive importance, while MSS identifies minimal sets of features sufficient for the prediction—highlighting that neither alone provides a complete view of model behavior.

ABSTRACT

For neural models to garner widespread public trust and ensure fairness, we must have human-intelligible explanations for their predictions. Recently, an increasing number of works focus on explaining the predictions of neural models in terms of the relevance of the input features. In this work, we show that feature-based explanations pose problems even for explaining trivial models. We show that, in certain cases, there exist at least two ground-truth feature-based explanations, and that, sometimes, neither of them is enough to provide a complete view of the decision-making process of the model. Moreover, we show that two popular classes of explainers, Shapley explainers and minimal sufficient subsets explainers, target fundamentally different types of ground-truth explanations, despite the apparently implicit assumption that explainers should look for one specific feature-based explanation. These findings bring an additional dimension to consider in both developing and choosing explainers.

Motivation & Objective

• To investigate the existence of multiple ground-truth feature-based explanations for model predictions, even in trivial models.
• To examine the fundamental differences between two major classes of explainers: Shapley values and minimal sufficient subsets (MSS).
• To challenge the implicit assumption that there is a single, unique ground-truth explanation for any model prediction.
• To highlight the limitations of each explainer type in fully capturing the decision-making process of a model.
• To advocate for greater transparency in specifying the type of explanation a method aims to provide, based on user needs and context.

Proposed method

• Uses rule-based models with explicit decision logic to analyze feature-based explanations, enabling full control and verification of model behavior.
• Computes Shapley values using the standard formula from coalitional game theory, which assigns each feature a contribution based on marginal contributions across all subsets.
• Identifies minimal sufficient subsets (MSS) as the smallest feature subsets that preserve the model’s prediction when used in isolation.
• Compares Shapley explanations and MSS explanations on identical model instances to contrast their interpretations of feature importance and necessity.
• Employs feature occlusion and baseline substitution to compute model outputs on subsets, ensuring consistent evaluation of MSS.
• Analyzes cases where multiple MSS exist (e.g., {"good", "very"} and {"nice"} in the same input), showing that explainers may miss or misrepresent such alternatives.

Experimental results

Research questions

• RQ1 Can multiple ground-truth feature-based explanations coexist for a single model prediction, even in simple, transparent models?
• RQ2 How do Shapley values and minimal sufficient subsets (MSS) differ in the type of explanation they aim to provide?
• RQ3 To what extent do Shapley values and MSS explainers each fail to fully represent the decision-making process of a model?
• RQ4 Can combining both types of explanations yield a more complete understanding of model behavior than either alone?
• RQ5 Why is it problematic to assume a single ground-truth explanation exists when multiple valid, distinct explanations may coexist?

Key findings

• For the same model prediction, multiple ground-truth feature-based explanations can exist, such as {"nice"} and {"good", "very"} in the same input, both of which are minimal sufficient subsets.
• Shapley values assign non-zero importance to features like "good" and "nice" in a way that reflects their marginal contribution, but do not indicate whether a feature is individually sufficient.
• Minimal sufficient subsets explainers may return only one subset (e.g., {"nice"}), even when multiple valid subsets exist, leading to incomplete or biased explanations.
• In cases with redundant or conflicting features (e.g., "amazing" and "bad" in a sentiment model), MSS explainers can produce artefacts that misrepresent the true decision logic.
• Shapley values can distinguish between features that are individually sufficient versus those that are redundant, such as showing "good" as informative even when "nice" alone suffices.
• Combining both Shapley and MSS explanations enables a more complete understanding—e.g., identifying that "good" is redundant when "nice" is present, but still informative in isolation.