[Paper Review] What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods

Julien Colin, Thomas Fel+2 more|arXiv (Cornell University)|Dec 6, 2021

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper proposes a human-centered evaluation framework for explainability methods using large-scale psychophysics experiments (n=1,150) across three real-world scenarios: bias detection, discovery of novel strategies, and failure case analysis. It finds that SmoothGrad is the most useful attribution method, while faithfulness metrics fail to predict human utility, suggesting explainability methods may need to convey 'what' features drive decisions, not just 'where'.

ABSTRACT

A multitude of explainability methods and associated fidelity performance metrics have been proposed to help better understand how modern AI systems make decisions. However, much of the current work has remained theoretical -- without much consideration for the human end-user. In particular, it is not yet known (1) how useful current explainability methods are in practice for more real-world scenarios and (2) how well associated performance metrics accurately predict how much knowledge individual explanations contribute to a human end-user trying to understand the inner-workings of the system. To fill this gap, we conducted psychophysics experiments at scale to evaluate the ability of human participants to leverage representative attribution methods for understanding the behavior of different image classifiers representing three real-world scenarios: identifying bias in an AI system, characterizing the visual strategy it uses for tasks that are too difficult for an untrained non-expert human observer as well as understanding its failure cases. Our results demonstrate that the degree to which individual attribution methods help human participants better understand an AI system varied widely across these scenarios. This suggests a critical need for the field to move past quantitative improvements of current attribution methods towards the development of complementary approaches that provide qualitatively different sources of information to human end-users.

Motivation & Objective

• To assess the practical usefulness of explainability methods in real-world scenarios from a human user perspective.
• To identify whether current faithfulness metrics accurately reflect the utility of attribution methods for end-users.
• To investigate whether perceptual similarity or explanation complexity predicts human performance in understanding model decisions.
• To determine if attribution methods can effectively support human understanding of model behavior in complex or biased decision contexts.
• To propose a human-centered evaluation framework that moves beyond theoretical metrics toward practical usability.

Proposed method

• Conducted large-scale psychophysics experiments with 1,150 participants across three real-world AI decision scenarios: bias detection, novel strategy discovery, and failure case understanding.
• Used human performance in prediction and decision-making tasks as the primary measure of explanation usefulness.
• Computed perceptual similarity scores between diagnostic image regions across classes to assess whether perceptual ambiguity affects explanation utility.
• Evaluated multiple attribution methods (e.g., Grad-CAM, SmoothGrad, Integrated Gradients) using both faithfulness metrics and human-annotated utility scores.
• Analyzed the correlation between faithfulness metrics, explanation complexity, perceptual similarity, and actual human performance to identify predictors of explanation failure.
• Released all data and code to enable reproducibility and future adoption of the human-centered evaluation framework.

Experimental results

Research questions

• RQ1 How useful are existing attribution methods in helping human users understand AI model decisions across real-world scenarios?
• RQ2 Do current faithfulness metrics reliably predict the practical usefulness of explainability methods for human users?
• RQ3 To what extent does the perceptual similarity between diagnostic image regions influence the effectiveness of attribution maps in aiding human understanding?
• RQ4 Can explanation complexity predict human performance in leveraging explanations to understand model decisions?
• RQ5 What role does the 'what' (semantic content) versus 'where' (spatial location) of explanations play in human interpretability?

Key findings

• SmoothGrad was the most useful attribution method across the tested scenarios, outperforming other methods in supporting human understanding.
• Faithfulness metrics showed no significant correlation with human-estimated utility, indicating they are poor predictors of practical usefulness.
• Perceptual similarity scores between diagnostic image regions were better predictors of explanation failure than faithfulness or complexity metrics.
• Explanation complexity had only a weak correlation with human performance, suggesting it is not a primary factor in explanation utility.
• When diagnostic features across classes were perceptually similar (e.g., cat vs. dog breeds), attribution methods failed to support human understanding, regardless of faithfulness or complexity.
• The results suggest that current attribution methods may be fundamentally limited in their ability to support human understanding because they do not communicate 'what' features are driving decisions, only 'where'.