[Paper Review] Visualizing Deep Neural Network Decisions: Prediction Difference Analysis

Luisa Zintgraf, Taco Cohen+2 more|arXiv (Cornell University)|Feb 15, 2017

Explainable Artificial Intelligence (XAI)Computer Science10 references431 citations

TL;DR

Introduces prediction difference analysis to visualize, for a given input, which image regions evidence a neural network’s decision for or against a class, with improvements over prior methods through conditional, multivariate sampling and deep visualization of hidden layers.

ABSTRACT

This article presents the prediction difference analysis method for visualizing the response of a deep neural network to a specific input. When classifying images, the method highlights areas in a given input image that provide evidence for or against a certain class. It overcomes several shortcoming of previous methods and provides great additional insight into the decision making process of classifiers. Making neural network decisions interpretable through visualization is important both to improve models and to accelerate the adoption of black-box classifiers in application areas such as medicine. We illustrate the method in experiments on natural images (ImageNet data), as well as medical images (MRI brain scans).

Motivation & Objective

• Motivate the need for interpretable deep learning in high-stakes domains like medicine and science.
• Develop an instance-specific visualization method that explains classifier decisions for individual inputs.
• Improve prior visualization approaches with conditional sampling, multivariate analysis, and deep visualization of hidden layers.

Proposed method

• Build on prediction difference analysis to quantify feature relevance for a class via WE_i(c|x) as the log-odds difference between p(c|x) and p(c|x_{ackslash i}).
• Introduce conditional sampling: approximate p(x_i|x_{ackslash i}) by p(x_i|x̂_{ackslash i}) where x̂_i is a patch surrounding x_i.
• Adopt multivariate sampling by evaluating patches k×k and averaging over overlapping patches for pixels, enabling more robust importance estimates.
• Extend the framework to deep networks by defining g(z|h_{ackslash i}) and AD_i(z|h) to assess hidden-layer contributions to deeper nodes.
• Provide an algorithm (Algorithm 1) to compute WE across patches with S samples.
• Demonstrate applicability on ImageNet DCNNs (AlexNet, GoogLeNet, VGG) and MRI diffusion-weighted images with a logistic regression classifier.

Experimental results

Research questions

• RQ1 How can we quantify and visualize the evidence a specific input provides for or against a given class in a DCNN?
• RQ2 Does conditional, multivariate sampling yield more precise and interpretable explanations than marginal, univariate approaches?
• RQ3 Can the method reveal how hidden-layer activations influence decisions in deeper layers or output nodes?
• RQ4 How does the visualization behave across different network architectures and data modalities (natural images vs medical images)?

Key findings

• Conditional sampling yields more targeted and fine-grained explanations than marginal sampling, concentrating evidence around the object.
• Multivariate, patch-based analysis provides meaningful, robust relevance maps compared to single-pixel removal.
• Visualizations at penultimate vs output layers reveal different class-discrimination cues, with output-layer visuals highlighting class-specific decisions among similar classes.
• The method exposes architecture-dependent differences in explanations across AlexNet, GoogLeNet, and VGG, e.g., context focus vs. object-specific cues.
• Applied to MRI data, the visualization aligns with expert intuition and contrasts with noise-prone weights, aiding clinical interpretation.