[Paper Review] Sanity Checks for Saliency Maps

Julius Adebayo, Justin Gilmer+4 more|arXiv (Cornell University)|Oct 8, 2018

Visual Attention and Saliency Detection604 citations

TL;DR

The paper presents an actionable methodology to evaluate what explanations saliency maps provide, showing that some methods can be independent of model or data, and thus inadequate for certain data- or model-sensitive tasks.

ABSTRACT

Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model. We interpret our findings through an analogy with edge detection in images, a technique that requires neither training data nor model. Theory in the case of a linear model and a single-layer convolutional neural network supports our experimental findings.

Motivation & Objective

• Motivate a practical evaluation framework for saliency explanations beyond visual inspection.
• Investigate whether saliency methods truly depend on the trained model and the data generating process.
• Identify tasks that saliency explanations can or cannot support, such as debugging and outlier detection.

Proposed method

• Proposes a methodology to test saliency methods by examining dependence on model and data.
• Conducts extensive experiments across saliency methods to assess sensitivity to model/data variations.
• Uses a linear model and a single-layer CNN to provide theoretical support for empirical findings.
• Draws an analogy with edge detection in images to interpret results.
• Reports updates to Guided Backprop experiments to fix a bug while preserving conclusions.

Experimental results

Research questions

• RQ1 Do saliency explanations depend on the specific model or data used to train it?
• RQ2 Are some saliency methods invariant to model or data changes, and what tasks does this affect?
• RQ3 Can saliency maps reliably reveal relationships between inputs and outputs for debugging or outlier detection?
• RQ4 Do theoretical insights from simple models align with empirical findings for complex models?

Key findings

• Some saliency methods are independent of both the model and the data generating process.
• Methods failing the proposed tests are inadequate for tasks sensitive to data or model.
• An edge-detection analogy helps interpret when explanations require minimal training data or models.
• Theoretical support from linear models and single-layer CNNs aligns with empirical results.
• Guided Backprop experiments were updated due to a bug, but overall conclusions remain unchanged.