[Paper Review] Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study

Ahmed Alqaraawi, Martin Schuessler+3 more|arXiv (Cornell University)|Feb 3, 2020

Explainable Artificial Intelligence (XAI)44 references30 citations

TL;DR

The study evaluates LRP-based saliency maps for CNN image classification and finds they aid users in learning salient features and improving task accuracy modestly (60.7% vs 55.1%), but do not substantially help predicting outputs on new images.

ABSTRACT

Convolutional neural networks (CNNs) offer great machine learning performance over a range of applications, but their operation is hard to interpret, even for experts. Various explanation algorithms have been proposed to address this issue, yet limited research effort has been reported concerning their user evaluation. In this paper, we report on an online between-group user study designed to evaluate the performance of "saliency maps" - a popular explanation algorithm for image classification applications of CNNs. Our results indicate that saliency maps produced by the LRP algorithm helped participants to learn about some specific image features the system is sensitive to. However, the maps seem to provide very limited help for participants to anticipate the network's output for new images. Drawing on our findings, we highlight implications for design and further research on explainable AI. In particular, we argue the HCI and AI communities should look beyond instance-level explanations.

Motivation & Objective

• Assess whether saliency maps improve lay and expert user understanding of CNN decisions in multi-label image classification.
• Investigate if visual explanations (LRP saliency maps) affect users' ability to predict CNN outputs on new images.
• Examine whether including classification scores alongside saliency maps changes user performance.
• Provide design implications for explainable AI by evaluating instance-level explanations beyond single predictions.

Proposed method

• Online between-group user study with a 2x2 factorial design.
• Participants predict CNN outcomes for 14 task images from PASCAL VOC hold-out set, and list 2–3 features the system is sensitive to and 2–3 it ignores.
• CNN model: VGG16 fine-tuned on PASCAL VOC 2012, with AP 0.91 train/val and 0.74 on PASCAL VOC 2007 test.
• Saliency maps generated with Layer-wise Relevance Propagation (LRP) using alpha-beta propagation (alpha=2, beta=1).
• Two independent variables: presence of saliency maps (yes/no) and presence of top-10 class scores (yes/no).
• Example images shown to participants chosen by cosine similarity in embedding space to task images, with varied outcomes (TP, FN, FP).

Experimental results

Research questions

• RQ1 Do saliency maps improve participants' ability to forecast CNN outputs on new images?
• RQ2 Does exposing classification scores alongside explanations influence user performance or trust?
• RQ3 What kinds of features do users mention when explaining CNN decisions, and how does this depend on saliency map availability?

Key findings

• Participants were more accurate at predicting CNN outcomes when saliency maps were shown (60.7% vs 55.1%), with p=0.045.
• Classification scores did not significantly affect prediction performance.
• Overall task accuracy remained relatively low, indicating explanations have limited utility for anticipating new predictions.
• Participants mentioned more Saliency-Features when saliency maps were present (83.9% vs 54.6%), indicating maps guided attention to highlighted regions.
• Confidence in forecasts did not differ across conditions and remained low.
• Saliency maps helped users learn about some image features the system is sensitive to but did not consistently improve understanding of the model’s generalization to new images.