[Paper Review] Natural Language Descriptions of Deep Visual Features

Evan Hernandez, Sarah Schwettmann+4 more|arXiv (Cornell University)|Jan 26, 2022

Cell Image Analysis TechniquesBiochemistry, Genetics and Molecular Biology18 citations

TL;DR

This paper introduces milan, a method that automatically generates fine-grained, natural language descriptions for individual neurons in deep neural networks by maximizing pointwise mutual information between neuron activation patterns and human-annotated image region descriptions. The approach enables detailed analysis, auditing, and editing of vision models, significantly improving robustness by identifying and removing neurons sensitive to spurious text features, reducing error rates by 12% on adversarial test sets.

ABSTRACT

Some neurons in deep networks specialize in recognizing highly specific perceptual, structural, or semantic features of inputs. In computer vision, techniques exist for identifying neurons that respond to individual concept categories like colors, textures, and object classes. But these techniques are limited in scope, labeling only a small subset of neurons and behaviors in any network. Is a richer characterization of neuron-level computation possible? We introduce a procedure (called MILAN, for mutual-information-guided linguistic annotation of neurons) that automatically labels neurons with open-ended, compositional, natural language descriptions. Given a neuron, MILAN generates a description by searching for a natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active. MILAN produces fine-grained descriptions that capture categorical, relational, and logical structure in learned features. These descriptions obtain high agreement with human-generated feature descriptions across a diverse set of model architectures and tasks, and can aid in understanding and controlling learned models. We highlight three applications of natural language neuron descriptions. First, we use MILAN for analysis, characterizing the distribution and importance of neurons selective for attribute, category, and relational information in vision models. Second, we use MILAN for auditing, surfacing neurons sensitive to human faces in datasets designed to obscure them. Finally, we use MILAN for editing, improving robustness in an image classifier by deleting neurons sensitive to text features spuriously correlated with class labels.

Motivation & Objective

• To develop a method for automatically labeling individual neurons in deep networks with expressive, open-ended natural language descriptions.
• To overcome limitations of prior neuron labeling techniques that rely on fixed, pre-defined label sets and cannot discover novel or complex neuron behaviors.
• To enable fine-grained characterization of neuron functions, including categorical, relational, and logical structures in visual features.
• To support practical applications in model interpretability, such as analyzing neuron importance, auditing for demographic sensitivity, and editing models to mitigate spurious correlations.
• To create a model-agnostic framework that works across diverse architectures, including CNNs and vision transformers, even when trained on data different from the annotation distribution.

Proposed method

• milan formulates neuron labeling as a search for a natural language description that maximizes pointwise mutual information (PMI) with the image regions where a neuron activates.
• It uses a newly collected dataset, milannotations, consisting of fine-grained human descriptions of image regions, to train models estimating p(description | exemplars) and p(description).
• For each neuron, milan identifies a representative set of activating image regions (exemplars), then searches for a description that achieves high PMI with these exemplars.
• The method is model-agnostic and can be applied to convolutional units in CNNs, fully connected units in vision transformers, and other layers across various architectures.
• The approach leverages learned models of language and visual feature distributions to generate descriptions that are both specific and compositional, capturing complex patterns in neuron behavior.
• The framework supports downstream tasks such as neuron analysis, auditing for sensitive features, and model editing by identifying and ablating neurons with specific, potentially harmful behaviors.

Experimental results

Research questions

• RQ1 Can we automatically generate rich, open-ended, and compositional natural language descriptions for individual neurons in deep networks?
• RQ2 Can such descriptions capture complex, fine-grained features like relational or logical structures in visual representations?
• RQ3 Can these descriptions be used to identify and mitigate spurious correlations in vision models, such as reliance on text features?
• RQ4 How effective is the method in auditing models for demographic sensitivity, especially when trained on anonymized data?
• RQ5 Can the generated descriptions guide meaningful model editing to improve robustness and generalization?

Key findings

• milan successfully identifies 300 text-related convolutional units in a ResNet18 model trained on a dataset with spurious text labels, confirming significant model capacity devoted to text recognition.
• Ablating just 13 of these text-selective neurons improves test accuracy on an adversarial test set by 4.9 percentage points, representing a 12% reduction in error rate.
• The improvement is specific to removing text-sensitive neurons: when neurons are sorted by importance and ablated in that order, accuracy improves by less than 1% if all neurons are considered, showing the method's precision in identifying harmful features.
• The generated descriptions show high agreement with human-generated descriptions across diverse model architectures and tasks, validating their interpretability and quality.
• The method enables effective auditing of models trained on anonymized data, surfacing neurons sensitive to human faces despite data obfuscation.
• The approach demonstrates practical utility in editing models to improve robustness, particularly in mitigating spurious correlations with text features that degrade performance on out-of-distribution data.