[Paper Review] Don't trust your eyes: on the (un)reliability of feature visualizations

Robert Geirhos, R. Zimmermann+3 more|arXiv (Cornell University)|Jun 7, 2023

Cell Image Analysis Techniques4 citations

TL;DR

This paper challenges the reliability of feature visualizations in neural networks by demonstrating they can be manipulated via 'fooling circuits' to display arbitrary, meaningless patterns while preserving model accuracy. The authors show that visualizations are processed differently than natural images, undermining their use for interpreting network behavior, and provide theoretical proof that only a tiny subset of functions can be reliably visualized—calling for structured network designs to improve interpretability.

ABSTRACT

How do neural networks extract patterns from pixels? Feature visualizations attempt to answer this important question by visualizing highly activating patterns through optimization. Today, visualization methods form the foundation of our knowledge about the internal workings of neural networks, as a type of mechanistic interpretability. Here we ask: How reliable are feature visualizations? We start our investigation by developing network circuits that trick feature visualizations into showing arbitrary patterns that are completely disconnected from normal network behavior on natural input. We then provide evidence for a similar phenomenon occurring in standard, unmanipulated networks: feature visualizations are processed very differently from standard input, casting doubt on their ability to "explain" how neural networks process natural images. This can be used as a sanity check for feature visualizations. We underpin our empirical findings by theory proving that the set of functions that can be reliably understood by feature visualization is extremely small and does not include general black-box neural networks. Therefore, a promising way forward could be the development of networks that enforce certain structures in order to ensure more reliable feature visualizations.

Motivation & Objective

• To investigate whether feature visualizations—widely used to interpret neural network behavior—can be systematically deceived.
• To develop a sanity check for feature visualizations based on differences in how they are processed compared to natural images.
• To provide theoretical evidence that only a very limited class of functions can be reliably visualized using standard methods.
• To motivate the development of neural network architectures that enforce structural inductive biases to ensure more trustworthy visualizations.

Proposed method

• Design and implement 'fooling circuits'—specialized network modules that redirect feature visualization outputs to arbitrary, unrelated patterns without affecting model accuracy on natural images.
• Train a modified Inception-V1 and ResNet-50 with these circuits to demonstrate that visualizations can be controlled at will.
• Perform a sanity check by comparing the processing paths of natural images and feature visualizations through the network, revealing divergent internal representations.
• Theoretical analysis proves that the set of functions whose internal representations can be reliably visualized via activation maximization is extremely small, even under mild assumptions.
• Use empirical comparisons to show that feature visualizations are not processed like natural inputs, suggesting they do not reflect the true inductive biases of the network.
• Propose that future interpretability research should focus on designing networks with enforced structural priors to improve visualization reliability.

Experimental results

Research questions

• RQ1 Can feature visualizations be intentionally manipulated to show arbitrary, unrelated patterns while maintaining model accuracy on natural data?
• RQ2 How do feature visualizations differ in their internal processing from natural images in standard neural networks?
• RQ3 What theoretical constraints limit the reliability of feature visualizations for interpreting general deep neural networks?
• RQ4 To what extent can a simple sanity check based on processing divergence detect unreliable visualizations?
• RQ5 Can the theoretical limitations of activation maximization be overcome by architectural design choices?

Key findings

• Feature visualizations can be manipulated via fooling circuits to display arbitrary patterns—such as the Mona Lisa—while the network maintains its original ImageNet accuracy.
• Even in unmodified networks, feature visualizations are processed along fundamentally different pathways than natural images, undermining their interpretability.
• Theoretical analysis proves that only a negligible fraction of neural network functions can be reliably visualized using standard activation maximization, even under mild assumptions.
• Empirical results show that feature visualizations and natural images activate different internal representations, indicating they do not reflect the same inductive biases.
• The proposed sanity check detects significant differences in processing, suggesting that visualizations may not be trustworthy for mechanistic interpretation.
• The findings imply that future interpretability methods must be built on architectures with enforced structural inductive biases to ensure reliable visualization.