[Paper Review] The effectiveness of feature attribution methods and its correlation with automatic evaluation scores

Giang V. Nguyen, Daeyoung Kim+1 more|arXiv (Cornell University)|May 31, 2021

Adversarial Robustness in Machine LearningComputer Science65 references27 citations

TL;DR

This study evaluates how feature attribution maps (GradCAM, EP, SOD) and nearest-neighbor explanations affect human-AI team performance on ImageNet and Stanford Dogs, including adversarial cases, and reveals weak correlation between automatic attribution metrics and actual human performance.

ABSTRACT

Explaining the decisions of an Artificial Intelligence (AI) model is increasingly critical in many real-world, high-stake applications. Hundreds of papers have either proposed new feature attribution methods, discussed or harnessed these tools in their work. However, despite humans being the target end-users, most attribution methods were only evaluated on proxy automatic-evaluation metrics (Zhang et al. 2018; Zhou et al. 2016; Petsiuk et al. 2018). In this paper, we conduct the first user study to measure attribution map effectiveness in assisting humans in ImageNet classification and Stanford Dogs fine-grained classification, and when an image is natural or adversarial (i.e., contains adversarial perturbations). Overall, feature attribution is surprisingly not more effective than showing humans nearest training-set examples. On a harder task of fine-grained dog categorization, presenting attribution maps to humans does not help, but instead hurts the performance of human-AI teams compared to AI alone. Importantly, we found automatic attribution-map evaluation measures to correlate poorly with the actual human-AI team performance. Our findings encourage the community to rigorously test their methods on the downstream human-in-the-loop applications and to rethink the existing evaluation metrics.

Motivation & Objective

• Assess whether attribution maps help humans improve image classification accuracy in a human-AI team setting.
• Compare attribution-based explanations with baseline explanations and nearest-neighbor prototypes across ImageNet and Stanford Dogs.
• Evaluate human-AI performance on real and adversarial images to understand robustness of explanations.
• Investigate the correlation between common automatic attribution metrics (Pointing Game, IoU, WSL) and actual human-AI performance.
• Compare lay users and ML experts on the usefulness of attribution maps versus nearest-neighbor explanations.

Proposed method

• Conduct a large-scale user study with lay participants and ML experts to judge whether the AI’s top-1 label is correct after seeing explanations.
• Use ResNet-34 pretrained on ImageNet as the classifier to generate predictions and explanations for ImageNet and Stanford Dogs.
• Compare six conditions: AI-only, Confidence, GradCAM, Extremal Perturbation (EP), Salient Object Detection (SOD), and 3-NN prototypes.
• Include real, misclassified, and adversarial images to test robustness of explanations.
• Evaluate attribution maps using Pointing Game, IoU, and weakly-supervised localization (WSL) against human judgments.
• Use 3-NN (nearest three training-set examples from the predicted class) as a prototype-based explanation.
• Recruit 320 participants (with 161 ImageNet and 159 Dogs qualified submissions) across two datasets.

Experimental results

Research questions

• RQ1 Do attribution maps (GradCAM, EP, SOD) improve human-AI team accuracy on ImageNet and Stanford Dogs compared to baselines?
• RQ2 How do 3-NN explanations compare to attribution maps in aiding human decisions across natural and adversarial images?
• RQ3 Are automatic attribution evaluation metrics (Pointing Game, IoU, WSL) predictive of actual human-AI team performance?
• RQ4 Do results differ between lay users and ML experts when using attribution maps or 3-NN explanations?
• RQ5 What is the effect of explanations on a harder fine-grained task (Stanford Dogs) versus a broad 1000-class dataset (ImageNet)?

Key findings

• 3-NN explanations outperform attribution maps across ImageNet and Stanford Dogs for lay users.
• On ImageNet, heatmaps plus confidence scores yield a 6–8% accuracy boost for human-AI teams, but only about half of this gain is due to heatmaps themselves.
• On Stanford Dogs, both 3-NN and heatmaps tend to hurt human-AI team accuracy compared to AI alone.
• In adversarial images, confidence scores alone help, and 3-NN provides a consistent ~4% gain over confidence alone.
• Automatic attribution metrics (Pointing Game, IoU, WSL) correlate poorly with actual human-AI team performance, with only a small positive correlation for GradCAM on ImageNet (IoU r=0.22, WSL r=0.15, Pointing Game r=0.21).
• ML experts using 3-NN significantly outperform those using GradCAM (mean accuracy 76.67% vs. 68.00%).