[Paper Review] Learning how to explain neural networks: PatternNet and PatternAttribution

Pieter Jan Kindermans, Kristof T. Schütt+5 more|arXiv (Cornell University)|May 16, 2017

Neural Networks and Applications246 citations

TL;DR

The paper analyzes explanation methods for neural networks in the linear setting, identifies shortcomings of existing approaches, and introduces PatternNet and PatternAttribution as theoretically sound, data-driven explanations that generalize to deep networks. It also provides empirical evidence on ImageNet (VGG-16) showing improved signal visualization and attributions.

ABSTRACT

DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple neural networks. We argue that explanation methods for neural nets should work reliably in the limit of simplicity, the linear models. Based on our analysis of linear models we propose a generalization that yields two explanation techniques (PatternNet and PatternAttribution) that are theoretically sound for linear models and produce improved explanations for deep networks.

Motivation & Objective

• Motivate reliable explanations by focusing on the linear model where signal and distractor can be controlled and analytically tracked.
• Show limitations of existing visualization/attribution methods (e.g., DeConvNet, Guided BackProp, LRP) in representing true signal.
• Propose PatternNet and PatternAttribution as theoretically sound, data-driven explanations for linear models and provide extensions to deep networks.
• Evaluate the proposed methods on practical networks and datasets to demonstrate qualitative and quantitative improvements.

Proposed method

• Model the input as x = s + d, with signal s = a_s y and distractor d, and analyze how linear weights w relate to the signal direction a_s.
• Introduce a quality criterion rho(S) that measures how well a signal estimator S removes information about the output y from the residuals, guiding the estimation of the true signal.
• Review existing signal estimators (S_x, S_w) and show their limitations in detecting the true signal, especially in nonlinear later layers.
• Derive PatternNet as a neuron's nonlinear signal estimator by optimizing the quality criterion, yielding a back-projection of the estimated signal to input space.
• Derive PatternAttribution as a root-point estimator for Deep Taylor Decomposition, yielding neuron-wise contributions to the classification score by ignoring the distractor.
• Provide practical estimators for positive/negative regimes with S_{a+-} to handle ReLU/nonlinear effects and derive closed-form solutions (Eq. 4.3 and Eq. 7).
• Demonstrate that PatternNet reduces to a gradient-like backpropagation with informative directions replacing raw weights during the backward pass.

Experimental results

Research questions

• RQ1 Do existing explanation methods reliably capture the true signal in linear models, and by extension in deep networks?
• RQ2 Can we define a principled quality criterion to separate signal from distractor in explanations?
• RQ3 How can we construct signal and attribution estimators (PatternNet/PatternAttribution) that outperform prior methods in both linear and nonlinear layers?
• RQ4 Do the proposed methods yield qualitatively and quantitatively clearer visualizations and heatmaps on real networks and datasets (e.g., ImageNet/VGG-16)?

Key findings

• Existing methods like DeConvNet and Guided BackProp do not consistently capture the true signal in linear models, motivating a signal-aware approach.
• A quality criterion rho(S) can guide learning of signal estimators that remove distractor information from residuals, improving explanations.
• PatternNet provides improved signal back-projections to input space, yielding crisper, more faithful visualizations than prior methods.
• PatternAttribution provides clearer neuron-wise attributions by ignoring distractors, acting as a root-point estimator for Deep Taylor Decomposition.
• Two-component estimator S_{a+-} accounts for positive/negative regimes due to ReLUs, improving signal estimation in dense layers.
• Applied to ImageNet with VGG-16, PatternNet/PatternAttribution produce better quantitative and qualitative explanations, with optimized estimators outperforming gradient-based or weight-based baselines in correlation and degradation tests.
• Theback-propagation-like computation for PatternAttribution uses replaced weights with informative directions, yielding faster explanations comparable to or faster than existing real-time saliency methods.