[Paper Review] Understanding Neural Networks Through Deep Visualization

Jason Yosinski, Jeff Clune+3 more|arXiv (Cornell University)|Jun 22, 2015

Generative Adversarial Networks and Image Synthesis10 references1,469 citations

TL;DR

This paper introduces two open-source tools to improve interpretability of deep neural networks: (1) real-time visualization of feature activations in response to live input, and (2) regularized optimization to generate interpretable visualizations of learned features. The key contribution is significantly clearer, more realistic feature visualizations using novel regularization techniques, enabling deeper insight into how convolutional neural networks detect objects and patterns.

ABSTRACT

Recent years have produced great advances in training large, deep neural networks (DNNs), including notable successes in training convolutional neural networks (convnets) to recognize natural images. However, our understanding of how these models work, especially what computations they perform at intermediate layers, has lagged behind. Progress in the field will be further accelerated by the development of better tools for visualizing and interpreting neural nets. We introduce two such tools here. The first is a tool that visualizes the activations produced on each layer of a trained convnet as it processes an image or video (e.g. a live webcam stream). We have found that looking at live activations that change in response to user input helps build valuable intuitions about how convnets work. The second tool enables visualizing features at each layer of a DNN via regularized optimization in image space. Because previous versions of this idea produced less recognizable images, here we introduce several new regularization methods that combine to produce qualitatively clearer, more interpretable visualizations. Both tools are open source and work on a pre-trained convnet with minimal setup.

Motivation & Objective

• To address the lack of interpretability in deep neural networks, especially in understanding intermediate layer representations.
• To develop interactive tools that help researchers and practitioners gain intuition about how trained convolutional neural networks (CNNs) respond to inputs.
• To improve the quality of feature visualization by introducing new regularization methods that produce more interpretable and realistic visualizations of learned features.
• To enable better debugging, model improvement, and transfer learning by revealing the true invariance and locality of learned features.
• To explore the potential of using discriminatively trained networks for generative modeling by leveraging learned priors and feature representations.

Proposed method

• The first tool visualizes feature activations in real time from each layer of a pre-trained CNN when processing live webcam input or static images.
• The second tool uses regularized gradient ascent in image space to optimize input images that maximally activate specific neurons or channels in the network.
• Novel regularization techniques—such as total variation, color jitter, and spatial smoothing—are introduced to constrain the optimization and produce more realistic, interpretable visualizations.
• The method combines multiple regularization terms to suppress noise and preserve structural coherence, resulting in clearer feature detectors (e.g., faces, wheels, eyes).
• The approach allows visualization of both the top activating training images and the optimized synthetic images that best represent the neuron’s preferred stimulus.
• The tools are implemented as open-source software and work with standard pre-trained models like AlexNet, requiring minimal setup.

Experimental results

Research questions

• RQ1 How can we improve the interpretability of intermediate feature representations in deep convolutional neural networks?
• RQ2 What kind of visualizations can be generated using regularized optimization that reveal meaningful, realistic features rather than noise?
• RQ3 How do feature detectors in trained CNNs respond to natural image parts, and are they localized or distributed across channels?
• RQ4 Can discriminatively trained networks be used to generate realistic images by enforcing both a prior on image space and class-specific activation?
• RQ5 To what extent do feature visualizations reveal invariance and hierarchical composition in deep networks?

Key findings

• The new regularization methods produce significantly clearer and more interpretable visualizations of learned features, such as faces, eyes, wheels, and fur patterns, compared to prior approaches.
• Feature detectors in deeper layers (e.g., conv5) respond to specific, natural parts like faces, wheels, and eyes, suggesting a degree of locality in representation.
• Some neurons respond to multiple object parts (e.g., a face detector also responds to flowers on a blanket), revealing complex, non-exclusive feature interactions.
• The visualizations reveal that higher layers learn increasingly complex and varied patterns, with a notable jump in diversity between the last convolutional layer and the first fully-connected layer.
• Even with simple hand-coded priors, the optimized images exhibit long-range spatial dependencies (e.g., beetles with features spanning over 100 pixels), indicating that discriminative models encode generative-like structure.
• The results suggest that discriminatively trained networks may be repurposed for generative modeling when combined with strong, learned priors on image space.