[Paper Review] Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods

Oana-Maria Camburu, Eleonora Giunchiglia+3 more|arXiv (Cornell University)|Oct 4, 2019

Explainable Artificial Intelligence (XAI)26 references36 citations

TL;DR

The paper distinguishes feature-additive and feature-selection explanations, and introduces an automatic verification framework for the feature-selection perspective using a non-trivial neural model, demonstrating failure modes of popular explainers and providing an open evaluation test.

ABSTRACT

For AI systems to garner widespread public acceptance, we must develop methods capable of explaining the decisions of black-box models such as neural networks. In this work, we identify two issues of current explanatory methods. First, we show that two prevalent perspectives on explanations --- feature-additivity and feature-selection --- lead to fundamentally different instance-wise explanations. In the literature, explainers from different perspectives are currently being directly compared, despite their distinct explanation goals. The second issue is that current post-hoc explainers are either validated under simplistic scenarios (on simple models such as linear regression, or on models trained on syntactic datasets), or, when applied to real-world neural networks, explainers are commonly validated under the assumption that the learned models behave reasonably. However, neural networks often rely on unreasonable correlations, even when producing correct decisions. We introduce a verification framework for explanatory methods under the feature-selection perspective. Our framework is based on a non-trivial neural network architecture trained on a real-world task, and for which we are able to provide guarantees on its inner workings. We validate the efficacy of our evaluation by showing the failure modes of current explainers. We aim for this framework to provide a publicly available, off-the-shelf evaluation when the feature-selection perspective on explanations is needed.

Motivation & Objective

• Highlight fundamental differences between feature-additive and feature-selection explanation perspectives for instance-wise explanations.
• Propose an automatic verification framework for evaluating feature-selection explainers on a non-trivial neural model with guarantees about the target model's behavior.
• Demonstrate failure modes of popular explainers (LIME, SHAP, L2X) on a real-world task and provide open-source evaluation tests.

Proposed method

• Utilize a RCNN-based model trained on a real-world beer-review task (BeerAdvocate) to create datasets where zero-contribution and clearly relevant tokens are identifiable.
• Prune data to eliminate handshakes and ensure at least one clearly relevant token per instance, yielding partitions S_x = SR_x ∪ SDK_x and N_x as zero-contribution tokens.
• Define evaluation metrics that penalize ranking zero-contribution tokens over clearly relevant tokens: %_first, %_misrnk, avg_misrnk.
• Evaluate three explainers (LIME, SHAP - feature-additive; L2X - feature-selection) under these metrics on three aspects (appearance, aroma, palate).
• Provide analysis of why LIME/SHAP outperformed L2X and discuss limitations of the framework (not a universal ground-truth test).
• Release the evaluation test for off-the-shelf use and discuss generality to other tasks such as computer vision.

Experimental results

Research questions

• RQ1 How do feature-additivity and feature-selection explanations differ in instance-wise explanations and evaluation běhavior.
• RQ2 Can we automatically verify the faithfulness of feature-selection explainers using a non-trivial neural network with guaranteed target-model behavior?
• RQ3 What are the failure modes of popular explainers (LIME, SHAP, L2X) when evaluated under a strict feature-selection framework on a real-world task?

Key findings

• LIME and SHAP frequently rank zero-contribution tokens higher than clearly relevant tokens, indicating failure modes under the feature-selection perspective.
• L2X often ranks several zero-contribution tokens above clearly relevant tokens, particularly when K (pre-specified feature count) is mismatched with the task; performance degrades when K is set to human-annotated averages.
• Overall, LIME and SHAP achieved lower error rates across most metrics for the tested aspects, while L2X showed higher misranking in several settings.
• The evaluation framework can automatically reveal critical failures without assuming ground-truth model rationales, and is adaptable to other domains.