[Paper Review] When Explanations Lie: Why Many Modified BP Attributions Fail

Leon Sixt, Maximilian Granz+1 more|arXiv (Cornell University)|Dec 20, 2019

Explainable Artificial Intelligence (XAI)47 references42 citations

TL;DR

The paper shows that most modified backprop attribution methods collapse to a single dominant direction (rank-1) across layers, making explanations largely independent of later-layer parameters, with DeepLIFT as a notable exception; it introduces the cosine similarity convergence (CSC) metric to diagnose this behavior.

ABSTRACT

Attribution methods aim to explain a neural network's prediction by highlighting the most relevant image areas. A popular approach is to backpropagate (BP) a custom relevance score using modified rules, rather than the gradient. We analyze an extensive set of modified BP methods: Deep Taylor Decomposition, Layer-wise Relevance Propagation (LRP), Excitation BP, PatternAttribution, DeepLIFT, Deconv, RectGrad, and Guided BP. We find empirically that the explanations of all mentioned methods, except for DeepLIFT, are independent of the parameters of later layers. We provide theoretical insights for this surprising behavior and also analyze why DeepLIFT does not suffer from this limitation. Empirically, we measure how information of later layers is ignored by using our new metric, cosine similarity convergence (CSC). The paper provides a framework to assess the faithfulness of new and existing modified BP methods theoretically and empirically. For code see: https://github.com/berleon/when-explanations-lie

Motivation & Objective

• Assess faithfulness of modified BP attribution methods across common architectures (VGG-16, ResNet-50) and datasets (CIFAR-10, ImageNet).
• Explain why numerous modified BP rules fail sanity checks and class-sensitivity tests.
• Provide theoretical and empirical tools to diagnose convergence to rank-1 matrices in backpropagation-based explanations.
• Offer guidance on when and how modified BP methods may be trusted or require revision.

Proposed method

• Theoretical analysis showing that the z+ rule yields a non-negative matrix product across layers that converges to a rank-1 matrix under assumptions.
• Definition and use of the cosine similarity convergence (CSC) metric to quantify convergence of relevance vectors across layers.
• Empirical evaluation on networks (VGG-16, ResNet-50) and datasets (CIFAR-10, ImageNet) using random logit and parameter-randomization sanity checks.
• Comparison of multiple modified BP methods (LRP variants, Deep Taylor Decomposition, PatternAttribution, DeepLIFT, Guided BP, Deconv, RectGrad) and identification of DeepLIFT as an exception.
• Analysis of PatternAttribution and PatternNet via singular value ratios to understand convergence behavior.
• Introduction of a DeepLIFT ablation variant that decouples positive and negative chains to show convergence properties.

Experimental results

Research questions

• RQ1 Do modified BP attribution methods produce explanations that depend on later-layer parameters?
• RQ2 Do these methods generically converge to a single dominant direction across layers, reducing class-sensitivity?
• RQ3 Why does DeepLIFT avoid the convergence to rank-1, and can insights from it improve other methods?
• RQ4 Can a metric like CSC reliably diagnose convergence and faithfulness of attribution methods?

Key findings

• Most modified BP methods (excluding DeepLIFT) converge to a rank-1 matrix, making explanations insensitive to later layers.
• The z+ rule and related methods yield a non-negative matrix chain whose product converges to a rank-1 matrix under the stated conditions.
• CSC effectively tracks how information from later layers is lost in the attribution chain across layers.
• DeepLIFT does not follow the same convergence and can avoid rank-1 collapse under its positive/negative separation rules.
• Negative relevance is identified as a crucial factor absent in several convergent methods, suggesting a potential path to improve class sensitivity.
• Across multiple architectures, converging methods produce saliency maps highly similar when last-layer parameters are altered or the logit is randomized.