[Paper Review] Neural Activation Patterns (NAPs): Visual Explainability of Learned Concepts

Alex Bäuerle, Daniel Jönsson+1 more|arXiv (Cornell University)|Jun 20, 2022

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper introduces Neural Activation Patterns (NAPs), a method that identifies layer-level concepts in deep neural networks by clustering entire activation distributions across neurons, rather than focusing on high activations alone. By grouping inputs with similar multi-unit activation profiles, NAPs reveal complex, distributed concepts that complement existing unit-level explainability techniques, enabling visual introspection of learned representations across diverse architectures and datasets.

ABSTRACT

A key to deciphering the inner workings of neural networks is understanding what a model has learned. Promising methods for discovering learned features are based on analyzing activation values, whereby current techniques focus on analyzing high activation values to reveal interesting features on a neuron level. However, analyzing high activation values limits layer-level concept discovery. We present a method that instead takes into account the entire activation distribution. By extracting similar activation profiles within the high-dimensional activation space of a neural network layer, we find groups of inputs that are treated similarly. These input groups represent neural activation patterns (NAPs) and can be used to visualize and interpret learned layer concepts. We release a framework with which NAPs can be extracted from pre-trained models and provide a visual introspection tool that can be used to analyze NAPs. We tested our method with a variety of networks and show how it complements existing methods for analyzing neural network activation values.

Motivation & Objective

• Address the limitation of existing explainability methods that focus only on high activation values, missing complex, distributed concepts in neural network layers.
• Develop a method to identify layer-level concepts by analyzing the full distribution of activation values across neurons, rather than individual units.
• Enable visual and interactive interpretation of learned representations through a framework that extracts and displays NAPs from pre-trained models.
• Facilitate model comparison across architectures by identifying consistent or divergent concept representations at the layer level.
• Complement existing explainability tools like feature visualization and activation maximization with a method sensitive to multi-unit interactions and distributed representations.

Proposed method

• Feed a set of inputs through a pre-trained neural network and extract the activation vector from a target layer.
• Normalize activation values on a per-unit basis to ensure equal contribution of each neuron to the clustering process.
• Apply a clustering algorithm (e.g., k-means or hierarchical clustering) to group inputs based on similarity in their full activation profiles across all units in the layer.
• Associate each cluster (NAP) with its corresponding input images, predictions, and labels for visual interpretation.
• Develop an interactive visualization interface to explore NAPs, including input images, activation distributions, and prediction metadata.
• Extend the method to various network architectures (e.g., ResNet50, Inception V3) and data types (e.g., MNIST, ImageNet) to demonstrate generalizability.

Experimental results

Research questions

• RQ1 Can clustering the full activation distribution across neurons reveal meaningful, interpretable concepts at the layer level that are missed by high-activation-only methods?
• RQ2 How do NAPs compare across different neural network architectures in terms of concept consistency and representation diversity?
• RQ3 To what extent can NAPs uncover biases or performance issues in model training data through visual inspection of activation clusters?
• RQ4 Can NAPs be used to identify and visualize hierarchical concept learning across multiple layers, such as the emergence of complex features like 'sulphur butterfly' from simpler ones?
• RQ5 How does the integration of NAPs with attribution methods improve the interpretability of which input features drive specific NAPs?

Key findings

• NAPs successfully identify layer-level concepts by clustering full activation profiles, revealing complex, distributed representations that are not captured by max-activation or feature visualization methods.
• The method demonstrates consistent concept discovery across different architectures—e.g., Inception V3 and ResNet50 show similar NAPs for high-level concepts like 'sulphur butterfly', indicating architectural convergence on similar representations.
• NAPs enable the detection of model weaknesses, such as bias or insufficient generalization, by revealing clusters of inputs with similar activation patterns but incorrect predictions.
• The framework supports model comparison across architectures by visualizing and contrasting NAPs, revealing both agreement and divergence in concept representation.
• NAPs provide a complementary view to existing explainability tools, particularly in capturing interdependencies among neurons that are invisible when analyzing individual units.
• The authors release a public framework and interactive visualization tool, enabling reproducible extraction and analysis of NAPs from pre-trained models across diverse datasets and network types.