[Paper Review] Compositional Explanations of Neurons

Jesse Mu, Jacob Andreas|arXiv (Cornell University)|Jun 24, 2020

Explainable Artificial Intelligence (XAI)38 references50 citations

TL;DR

The paper introduces a method to explain individual neurons by composing logical concepts into explanations, revealing both perceptual abstractions and shallow heuristics in vision and NLP neurons, and showing how these explanations relate to performance and can guide targeted adversarial perturbations.

ABSTRACT

We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts that closely approximate neuron behavior. Compared to prior work that uses atomic labels as explanations, analyzing neurons compositionally allows us to more precisely and expressively characterize their behavior. We use this procedure to answer several questions on interpretability in models for vision and natural language processing. First, we examine the kinds of abstractions learned by neurons. In image classification, we find that many neurons learn highly abstract but semantically coherent visual concepts, while other polysemantic neurons detect multiple unrelated features; in natural language inference (NLI), neurons learn shallow lexical heuristics from dataset biases. Second, we see whether compositional explanations give us insight into model performance: vision neurons that detect human-interpretable concepts are positively correlated with task performance, while NLI neurons that fire for shallow heuristics are negatively correlated with task performance. Finally, we show how compositional explanations provide an accessible way for end users to produce simple "copy-paste" adversarial examples that change model behavior in predictable ways.

Motivation & Objective

• Identify whether neurons learn meaningful compositional concepts or polysemantic/unrelated activations in vision and NLP.
• Assess how compositional explanations relate to model performance across tasks.
• Demonstrate whether neuron explanations enable predictable manipulation of model behavior through targeted inputs.

Proposed method

• Define a space of primitive concepts as binary detectors over inputs (e.g., image regions, lexical features).
• Convert neuron activations to binary masks and measure similarity via IoU with concepts.
• Extend to compositional logic forms using Or, And, Not, and Neighbors operators to create L(C).
• Search for best explanations L in the compositional space by beam search with a length limit (N) to maximize IoU.
• Apply the method to image classification (ResNet-18 on Places365 with Broden annotations) and natural language inference (SNLI using a BiLSTM + MLP, probing premise/hypothesis words and overlaps).
• Compare explanation quality (IoU) at different formula lengths to assess compositionality and interpretability.

Experimental results

Research questions

• RQ1 What kinds of compositional concepts do neurons learn in vision and NLP models?
• RQ2 How does the interpretability of neuron explanations relate to model accuracy across tasks?
• RQ3 Can compositional neuron explanations be used to predict and influence model behavior through targeted inputs?

Key findings

• Neurons often learn compositional concepts; some detectors capture meaningful abstractions, while others are polysemantic or rely on unrelated features.
• IoU-based explanation quality improves with longer formula length, rising from a mean 0.059 at N=1 to 0.099 at N=10 in image classification (significant increase; p=2e-9).
• In vision, more interpretable neurons (higher IoU) correlate with higher accuracy when the neuron is active (r=0.31, p<1e-13), with stronger correlation as formula length grows.
• In NLI, explanations often reflect shallow lexical heuristics (e.g., gender, overlap) and, paradoxically, more interpretable neurons tend to anticorrelate with task performance (r=-0.60, p<1e-08), suggesting many explanations capture spurious correlations.
• The approach enables creating “copy-paste” adversarial examples by targeting explanatory neurons, demonstrating predictable, model-influencing perturbations in both vision and NLI.