[Paper Review] A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning

Sina Mohseni, Eric D. Ragan+1 more|arXiv (Cornell University)|Jan 16, 2018

Explainable Artificial Intelligence (XAI)30 references46 citations

TL;DR

Introduces a human-attention baseline benchmark to quantitatively evaluate model saliency explanations, comparing multi-layer human-attention masks against segmentation masks and human judgments using Grad-CAM and LIME explanations.

ABSTRACT

Research in interpretable machine learning proposes different computational and human subject approaches to evaluate model saliency explanations. These approaches measure different qualities of explanations to achieve diverse goals in designing interpretable machine learning systems. In this paper, we propose a human attention benchmark for image and text domains using multi-layer human attention masks aggregated from multiple human annotators. We then present an evaluation study to evaluate model saliency explanations obtained using Grad-cam and LIME techniques. We demonstrate our benchmark's utility for quantitative evaluation of model explanations by comparing it with human subjective ratings and ground-truth single-layer segmentation masks evaluations. Our study results show that our threshold agnostic evaluation method with the human attention baseline is more effective than single-layer object segmentation masks to ground truth. Our experiments also reveal user biases in the subjective rating of model saliency explanations.

Motivation & Objective

• Motivate the need for objective, human-grounded evaluation of local explanation methods.
• Propose a multi-layer human-attention benchmark as a ground truth for saliency explanations.
• Enable fast, reproducible, and objective evaluation of saliency methods across image and text domains.

Proposed method

• Collect multi-layer human-attention masks from 10 annotators per sample for images and text.
• Aggregate annotations via union of regions/words to form multi-layer masks.
• Compare Grad-CAM and LIME saliency maps against three baselines using pixel-wise MAE without thresholding.
• For images, optionally mask out background using exact segmentation masks to isolate target pixels.
• Conduct human-subject studies to compare ground-truth baselines with subjective ratings.
• Analyze FP and FN errors separately to understand types of explanation errors.

Experimental results

Research questions

• RQ1 Does the multi-layer human-attention baseline provide objective ground truth that differs from single-layer segmentation masks?
• RQ2 How do Grad-CAM and LIME saliency maps score against different ground-truth baselines (human-attention vs segmentation) and against human judgments?
• RQ3 What biases exist in subjective ratings of saliency explanations, and how do FP vs FN errors influence ratings?
• RQ4 What is the relationship between ground-truth baselines and subjective ratings in terms of agreement or discrepancy?

Key findings

• The human-attention baseline correlated with segmentation-based ground truth but differed in FN error handling, indicating higher granularity and potential false negatives.
• Subjective human ratings differ from both ground-truth baselines, showing measurable biases in explanations and evaluation.
• Ground-truth with human-attention masks yielded threshold-agnostic MAE evaluations that do not require binary mask generation, offering granularity beyond single-layer masks.
• LIME explanations elicited different user responses than Grad-CAM explanations, revealing influence of visual appearance on judgments.
• Users showed less sensitivity to FP errors and more to FN errors, suggesting incomplete foreground coverage affects perceived explanation quality.
• The study demonstrates that objective ground-truth baselines and subjective judgments can diverge, highlighting the need for complementary evaluation approaches.