[Paper Review] The (Un)reliability of saliency methods

Pieter-Jan Kindermans, Sara Hooker+6 more|arXiv (Cornell University)|Nov 2, 2017

Explainable Artificial Intelligence (XAI)Computer Science163 citations

TL;DR

The paper shows that many saliency methods fail input invariance: simple input shifts can change attributions without affecting the model, and only some reference-point methods (like PatternAttribution) can be invariant under certain transformations.

ABSTRACT

Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step ---adding a constant shift to the input data--- to show that a transformation with no effect on the model can cause numerous methods to incorrectly attribute. In order to guarantee reliability, we posit that methods should fulfill input invariance, the requirement that a saliency method mirror the sensitivity of the model with respect to transformations of the input. We show, through several examples, that saliency methods that do not satisfy input invariance result in misleading attribution.

Motivation & Objective

• Motivate reliability in saliency methods by defining input invariance as essential for trustworthy explanations.
• Demonstrate that constant input shifts can alter attribution despite identical model predictions.
• Evaluate how different saliency categories (gradients, signal methods, attribution methods) behave under input shifts.
• Highlight the role of reference points and data normalization in achieving invariance.

Proposed method

• Define input invariance as mirroring the model's sensitivity to input transformations.
• Experiment with MNIST to compare saliency heatmaps from various methods on input pairs related by a constant shift.
• Evaluate gradient, signal, and attribution methods (GI, IG, DTD) for input invariance under different references.
• Analyze how different reference points (zero, black image, PA baseline, LRP) affect attribution invariance.
• Show how data normalization can help some methods satisfy input invariance.

Experimental results

Research questions

• RQ1 Do common saliency methods satisfy input invariance under simple input transformations like constant shifts?
• RQ2 How do gradient, signal, and attribution-based methods compare in terms of input invariance?
• RQ3 What is the impact of reference point choice on the reliability of attribution methods such as Integrated Gradients and Deep Taylor Decomposition?
• RQ4 Can data normalization mitigate failures in input invariance for saliency methods?

Key findings

• Numerous saliency methods do not satisfy input invariance when inputs are shifted by a constant vector, altering attributions without changing predictions.
• Gradient and signal methods (including raw gradients, PatternNet, and Guided Backprop) can be invariant when comparing networks with identical weights, but attribution methods may fail depending on the reference point.
• Gradient times input is sensitive to mean shifts and does not satisfy input invariance.
• Integrated Gradients and Deep Taylor Decomposition depend on the chosen reference point, and invariance is achieved only for certain references and transformations.
• PatternAttribution consistently satisfies input invariance under the considered shifts because it incorporates data covariance.
• SmoothGrad inherits the sensitivity of the underlying method and may fail invariance for some references, but can be invariant with PA or black-image references.