[Paper Review] Visual Explanations for Convolutional Neural Networks via Input Resampling.

Benjamin J. Lengerich, Sandeep Konam+3 more|arXiv (Cornell University)|Jul 30, 2017

Explainable Artificial Intelligence (XAI)1 references16 citations

TL;DR

This paper proposes a framework for interpreting convolutional neural networks by analyzing internal feature representations through input resampling. By measuring correlation between neuron activations and network output, and activation precision across perturbed inputs, the method identifies neurons responsible for predictive influence and generalizable features, revealing insights into internal attention mechanisms.

ABSTRACT

The predictive power of neural networks often costs model interpretability. Several techniques have been developed for explaining model outputs in terms of input features; however, it is difficult to translate such interpretations into actionable insight. Here, we propose a framework to analyze predictions in terms of the model's internal features by inspecting information flow through the network. Given a trained network and a test image, we select neurons by two metrics, both measured over a set of images created by perturbations to the input image: (1) magnitude of the correlation between the neuron activation and the network output and (2) precision of the neuron activation. We show that the former metric selects neurons that exert large influence over the network output while the latter metric selects neurons that activate on generalizable features. By comparing the sets of neurons selected by these two metrics, our framework suggests a way to investigate the internal attention mechanisms of convolutional neural networks.

Motivation & Objective

• To improve interpretability of convolutional neural networks by analyzing internal feature representations rather than relying solely on input-level explanations.
• To identify neurons that exert significant influence on model predictions through correlation with output changes under input perturbations.
• To distinguish neurons activated on generalizable features using precision of activation across perturbed inputs.
• To provide actionable insights into internal attention mechanisms by comparing neurons selected via influence and generalization metrics.

Proposed method

• Perturb the input image to generate a set of resampled inputs for analysis.
• Measure the correlation between neuron activation and network output across the perturbed inputs to assess predictive influence.
• Compute the precision of neuron activation—how consistently neurons fire across perturbations—to identify generalizable feature detectors.
• Select neurons based on high correlation (influence) and high precision (generalization) to compare their roles in the network.
• Use the intersection and contrast of these neuron sets to infer internal attention mechanisms.
• Apply the framework to trained CNNs on image classification tasks to analyze feature-level interpretability.

Experimental results

Research questions

• RQ1 Which neurons in a CNN are most influential in determining the final prediction?
• RQ2 Which neurons respond to features that generalize across input variations?
• RQ3 How do neurons with high predictive influence differ from those detecting generalizable features?
• RQ4 Can comparing influence and precision metrics reveal insights into internal attention mechanisms?

Key findings

• Neurons with high correlation between activation and output are strongly influential in shaping the network’s final prediction.
• Neurons with high activation precision across perturbed inputs are more likely to respond to generalizable, robust features in the input.
• The overlap and distinction between influence-based and precision-based neuron sets reveal how internal attention is distributed across feature detectors.
• The framework enables a more actionable interpretation of model behavior by linking neuron activity to both predictive power and feature generalization.