[Paper Review] Latent SHAP: Toward Practical Human-Interpretable Explanations

Ron Bitton, Alon Malach+6 more|arXiv (Cornell University)|Nov 27, 2022

Explainable Artificial Intelligence (XAI)5 citations

TL;DR

Latent SHAP introduces a model-agnostic framework that generates locally faithful, human-interpretable explanations for complex models without requiring a fully invertible transformation from input features to interpretable concepts. By leveraging a latent space of human-understandable features and approximating Shapley values in this space, it enables intuitive, high-level explanations even when direct invertibility is infeasible, outperforming standard SHAP in interpretability on celebrity attractiveness classification.

ABSTRACT

Model agnostic feature attribution algorithms (such as SHAP and LIME) are ubiquitous techniques for explaining the decisions of complex classification models, such as deep neural networks. However, since complex classification models produce superior performance when trained on low-level (or encoded) features, in many cases, the explanations generated by these algorithms are neither interpretable nor usable by humans. Methods proposed in recent studies that support the generation of human-interpretable explanations are impractical, because they require a fully invertible transformation function that maps the model's input features to the human-interpretable features. In this work, we introduce Latent SHAP, a black-box feature attribution framework that provides human-interpretable explanations, without the requirement for a fully invertible transformation function. We demonstrate Latent SHAP's effectiveness using (1) a controlled experiment where invertible transformation functions are available, which enables robust quantitative evaluation of our method, and (2) celebrity attractiveness classification (using the CelebA dataset) where invertible transformation functions are not available, which enables thorough qualitative evaluation of our method.

Motivation & Objective

• To address the limitation of existing model-agnostic explanation methods like SHAP and LIME, which produce explanations in terms of low-level or encoded input features that are hard for humans to interpret.
• To develop a framework that generates explanations in terms of high-level, human-understandable features (e.g., 'smiling', 'wearing earrings') without requiring a fully invertible mapping from input features to these concepts.
• To ensure faithfulness of explanations to the original model’s behavior while improving interpretability in real-world scenarios where invertible transformations are impractical.
• To evaluate the method both quantitatively (using invertible transformations) and qualitatively (in non-invertible settings) on a celebrity attractiveness classification task.

Proposed method

• The framework operates in a black-box manner, using a pre-trained model and a separate, human-interpretable feature extractor to map inputs into a latent space of high-level concepts.
• It applies a variant of the SHAP algorithm to compute feature attributions in the interpretable latent space, approximating Shapley values efficiently using a background dataset.
• The method does not require the transformation from raw inputs to interpretable features to be invertible, enabling use in scenarios where such a mapping is ambiguous or infeasible.
• It uses a background dataset of samples to estimate the expected contribution of each interpretable feature, enabling local explanation via perturbation-based approximation.
• The approach supports both local and global explanation generation, with feature importance scores visualized per instance and aggregated across the dataset.
• It integrates concept-based explanations by defining a set of disentangled, human-readable features (e.g., 'arched eyebrows', 'high cheekbones') derived from the input image.

Experimental results

Research questions

• RQ1 Can Latent SHAP generate human-interpretable explanations that are both locally faithful and intuitive when the transformation from input features to interpretable features is not invertible?
• RQ2 How do the explanations from Latent SHAP compare qualitatively to standard SHAP in terms of interpretability and meaningfulness on real-world image classification tasks?
• RQ3 To what extent do the global feature attributions from Latent SHAP align with ground-truth correlations between human-interpretable features and model predictions?
• RQ4 How sensitive are the explanations to the size of the background dataset used for approximation?

Key findings

• In the controlled experiment with invertible transformations, Latent SHAP’s explanations were qualitatively more intuitive than standard SHAP, enabling users to identify high-level features such as 'smiling' and 'wearing earrings' more easily.
• On the CelebA dataset, Latent SHAP produced explanations that clearly distinguished between attractive and unattractive faces, assigning high positive importance to a few key features (e.g., 'smiling') in attractive faces and low or negative importance to many features in unattractive ones.
• The global feature attributions from Latent SHAP closely matched ground-truth correlations: for example, 'Heavy Makeup' showed high positive contribution for high values and low for low values, reflecting its known correlation with attractiveness.
• The male feature, which is inversely correlated with attractiveness, was correctly assigned negative attributions by Latent SHAP, confirming its faithfulness to model behavior.
• The size of the background dataset significantly affected feature importance: for instance, 'high cheekbones' dropped from the 7th most important feature (importance = -0.04) to outside the top-9 when the background size increased from 50 to 150 samples.
• Latent SHAP demonstrated robustness and interpretability in non-invertible settings, where traditional SHAP fails due to pixel-level explanations lacking semantic meaning.