[Paper Review] On the overlooked issue of defining explanation objectives for local-surrogate explainers

Rafael Poyiadzi, Xavier Renard+3 more|arXiv (Cornell University)|Jun 10, 2021

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper identifies a critical gap in local-surrogate explainability methods: despite sharing a common goal of explaining individual model predictions, they lack clarity in defining their explanation objectives, leading to inconsistent and incomparable outputs. The authors analyze neighborhood construction strategies across methods like LIME, KernelSHAP, and GSLS, revealing divergent information extraction from black-box models, which undermines comparability and user trust in explainability tools.

ABSTRACT

Local surrogate approaches for explaining machine learning model predictions have appealing properties, such as being model-agnostic and flexible in their modelling. Several methods exist that fit this description and share this goal. However, despite their shared overall procedure, they set out different objectives, extract different information from the black-box, and consequently produce diverse explanations, that are -- in general -- incomparable. In this work we review the similarities and differences amongst multiple methods, with a particular focus on what information they extract from the model, as this has large impact on the output: the explanation. We discuss the implications of the lack of agreement, and clarity, amongst the methods' objectives on the research and practice of explainability.

Motivation & Objective

• To identify and analyze the lack of clarity in explanation objectives among local-surrogate explainers.
• To investigate how different neighborhood strategies in local surrogate methods extract varying information from black-box models.
• To highlight the consequences of this ambiguity on method comparability and practical deployment of explainability.
• To advocate for user-centric specification of interpretability objectives before selecting or designing explainability methods.
• To call for formal, well-defined objectives in explainability research to improve method transparency and reliability.

Proposed method

• The paper conducts a theoretical analysis of neighborhood construction in local-surrogate explainers, focusing on how each method samples and weights instances around the prediction to explain.
• It compares neighborhood strategies across LIME, GSLS, LEAP, and KernelSHAP, emphasizing differences in sampling distributions and weighting schemes.
• The authors use a shared experimental setup (Half-Moons dataset and a neural network black-box) to visualize and contrast the neighborhoods generated by each method.
• They formalize the information extraction process in KernelSHAP, showing reliance on Shapley value approximations and assumptions like feature independence.
• The study evaluates explanations from LIME and KernelSHAP on the Adult dataset, illustrating conflicting feature importance rankings for the same prediction.
• It critiques the heuristic and approximative nature of current methods, arguing that outputs lack a clear semantic link to formal objectives.

Experimental results

Research questions

• RQ1 How do different local-surrogate explainers define their explanation objectives, and to what extent are these objectives transparent or consistent?
• RQ2 What specific information do various local-surrogate methods extract from the black-box model, and how does this affect the resulting explanations?
• RQ3 Why do methods like LIME and KernelSHAP produce conflicting explanations for the same prediction, even with identical inputs and models?
• RQ4 To what extent do current local-surrogate methods rely on approximations and assumptions that compromise the interpretability of their outputs?
• RQ5 How can the field of explainable AI improve by formalizing explanation objectives before method development?

Key findings

• Different local-surrogate methods generate significantly different neighborhoods around the same instance, even when using the same black-box model and prediction target.
• LIME uses a Gaussian neighborhood with locality-based weighting, while GSLS samples uniformly around the nearest counterfactual point, leading to distinct local representations.
• KernelSHAP extracts information via Shapley value approximations, relying on feature independence and linearity assumptions, which may distort the true contribution of features.
• Despite sharing the same high-level goal of explaining predictions, methods like LIME and KernelSHAP produce conflicting feature importance rankings on the same instance in the Adult dataset.
• The lack of formal, user-driven explanation objectives leads to incomparable outputs, hindering method evaluation and trust in real-world applications.
• The current state of the art in local surrogate explainers suffers from a 'solution-first, problem-later' approach, undermining the reliability and comparability of explanations.