[Paper Review] exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models

Benjamin Hoover, Hendrik Strobelt+1 more|arXiv (Cornell University)|Oct 11, 2019

Topic Modeling18 references47 citations

TL;DR

exBERT is an interactive visualization tool that analyzes both attention patterns and token embeddings in Transformer models, enabling users to inspect what learned representations encode by matching input contexts to a large annotated corpus. It integrates attention views with corpus-driven nearest-neighbor searches to reveal linguistic information captured by heads and embeddings.

ABSTRACT

Large language models can produce powerful contextual representations that lead to improvements across many NLP tasks. Since these models are typically guided by a sequence of learned self attention mechanisms and may comprise undesired inductive biases, it is paramount to be able to explore what the attention has learned. While static analyses of these models lead to targeted insights, interactive tools are more dynamic and can help humans better gain an intuition for the model-internal reasoning process. We present exBERT, an interactive tool named after the popular BERT language model, that provides insights into the meaning of the contextual representations by matching a human-specified input to similar contexts in a large annotated dataset. By aggregating the annotations of the matching similar contexts, exBERT helps intuitively explain what each attention-head has learned.

Motivation & Objective

• Provide an interactive, model-agnostic visualization to inspect attention and token embeddings in Transformer models.
• Enable exploration of what linguistic and positional information learned representations encode via corpus-based nearest-neighbor searches.
• Allow users to compare attention patterns and embedding similarities to reveal how heads and layers capture metadata (POS, DEP, NER, etc.).
• Demonstrate applicability through a BERT case study on the Wizard of Oz corpus to show progression of linguistic feature encoding across layers.

Proposed method

• Three main components: Attention View (interactive layer/head selection and masking), Corpus View (nearest-neighbor search over an annotated reference corpus), and Summary View (metadata histograms).
• Nearest-neighbor search uses cosine similarity over token embeddings and over concatenated head embeddings to find top matches in a linguistically annotated corpus.
• Head embeddings are defined by concatenating normalized head representations within a layer, enabling selective searches over subsets of heads.
• The reference corpus is sentence-split and annotated with linguistic metadata (POS, DEP, NER) to provide meaningful context for matches.
• Search results display top matches with metadata on hover, and summaries are provided to illustrate the linguistic features captured by the embeddings.
• The approach can be applied to different Transformer models and corpora beyond BERT by adapting tokenization and metadata assignments.

Experimental results

Research questions

• RQ1 What linguistic and Structural information do learned token embeddings encode across different layers of Transformer models?
• RQ2 How do attention heads contribute to encoding linguistic features, and can their roles be interpreted via corpus-based nearest-neighbor searches?
• RQ3 Can interactive visualization reveal the progression of linguistic feature encoding (e.g., POS, DEP, NER) across model layers and heads?
• RQ4 Is the tool effective in explaining what masked tokens represent by linking embeddings and attention to annotated corpus context?

Key findings

• Attention and embedding analyses reveal progressive encoding of linguistic features as layers increase (e.g., from punctuation/DET at early layers to verbs at later layers in the masked token scenario).
• Searching by head embeddings can expose dependency structures (e.g., DOBJ, ROOT, AUX) that may rely on combinations of heads rather than a single head.
• Positional heads show systematic attention to following words and reveal corresponding POS/DEP patterns, illustrating how some heads encode positional information.
• The tool demonstrates that a combination of heads, rather than any single head, detects certain linguistic relationships, highlighting the distributed nature of learned representations.
• exBERT provides a practical, model-agnostic way to inspect what contextual representations learn by integrating attention visualization with corpus-based metadata annotations.