[Paper Review] How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels

Hua Shen, Ting-Hao Huang|arXiv (Cornell University)|Aug 26, 2020

Explainable Artificial Intelligence (XAI)16 references4 citations

TL;DR

This study evaluates whether machine-generated visual interpretations (e.g., saliency maps) help general users infer why deep neural networks incorrectly classify images. Using crowdsourced experiments on 200 ImageNet images, it finds that interpretations significantly reduce human accuracy in guessing incorrect model predictions by ~10%, suggesting they may mislead rather than clarify model errors.

ABSTRACT

Explaining to users why automated systems make certain mistakes is important and challenging. Researchers have proposed ways to automatically produce interpretations for deep neural network models. However, it is unclear how useful these interpretations are in helping users figure out why they are getting an error. If an interpretation effectively explains to users how the underlying deep neural network model works, people who were presented with the interpretation should be better at predicting the model's outputs than those who were not. This paper presents an investigation on whether or not showing machine-generated visual interpretations helps users understand the incorrectly predicted labels produced by image classifiers. We showed the images and the correct labels to 150 online crowd workers and asked them to select the incorrectly predicted labels with or without showing them the machine-generated visual interpretations. The results demonstrated that displaying the visual interpretations did not increase, but rather decreased, the average guessing accuracy by roughly 10%.

Motivation & Objective

• To assess whether machine-generated visual interpretations improve general users' ability to reason about deep neural network errors in image classification.
• To investigate whether interpretations help users identify why a model made a wrong prediction, especially in cases of misclassification.
• To evaluate the effectiveness of post-hoc interpretations (e.g., saliency maps) in supporting non-expert users in understanding model failures.
• To determine under which conditions interpretations are helpful or harmful in aiding human inference about model errors.

Proposed method

• Conducted two controlled experiments using Amazon Mechanical Turk with 150 crowd workers to evaluate human performance in guessing incorrectly predicted labels.
• Presented each worker with an image and the correct label, informing them the model had made an error, with or without a machine-generated saliency map interpretation.
• Used a within-subjects design where each image was shown in both 'with interpretation' ([Int]) and 'without interpretation' ([No-Int]) conditions to control for image variability.
• Collected 2,000 total submissions (1,000 per condition), with 42 workers in the [Int] group and 63 in the [No-Int] group, ensuring no overlap between conditions.
• Measured human accuracy in selecting the correct incorrect label from five options (four distractors) to assess the impact of interpretations.
• Performed paired t-tests to evaluate statistical significance of differences in guessing accuracy between [Int] and [No-Int] conditions.

Experimental results

Research questions

• RQ1 Does showing machine-generated visual interpretations improve human users' ability to guess the incorrectly predicted label of an image?
• RQ2 Are there specific types of misclassification errors where interpretations are more or less effective?
• RQ3 Does the presence of interpretations lead to a statistically significant change in human guessing accuracy?
• RQ4 Under what conditions do interpretations hinder rather than help users understand model errors?

Key findings

• The average human guessing accuracy was 0.63 in the [Int] condition and 0.73 in the [No-Int] condition, resulting in a statistically significant decrease of 10% when interpretations were shown (p < 0.01).
• For category C3 (similar-looking objects), the accuracy dropped from 0.84 (no interpretation) to 0.66 (with interpretation), indicating interpretations worsened performance.
• For category C4 (errors due to background or scene confusion), accuracy decreased from 0.59 to 0.41 when interpretations were shown, showing a significant negative impact.
• In contrast, for C1 and C2 categories, accuracy slightly increased with interpretations (C1: 0.57 vs. 0.52; C2: 0.74 vs. 0.71), but differences were not statistically significant.
• The overall results indicate that visual interpretations did not improve, and in fact reduced, human performance in inferring model misclassifications.
• The findings suggest that interpretations may mislead users, particularly in cases involving visual similarity or spurious correlations in the image background.