[Paper Review] On Identifiability in Transformers

Gino Brunner, Yang Liu+4 more|arXiv (Cornell University)|Aug 12, 2019

Topic Modeling48 references76 citations

TL;DR

The paper analyzes identifiability of attention weights and token embeddings in Transformers, introduces effective attention and Hidden Token Attribution, and shows substantial but not complete token identity preservation with depth due to context mixing.

ABSTRACT

In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the aggregation of context into hidden tokens. We show that, for sequences longer than the attention head dimension, attention weights are not identifiable. We propose effective attention as a complementary tool for improving explanatory interpretations based on attention. Furthermore, we show that input tokens retain to a large degree their identity across the model. We also find evidence suggesting that identity information is mainly encoded in the angle of the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based on gradient attribution. Overall, we show that self-attention distributions are not directly interpretable and present tools to better understand and further investigate Transformer models.

Motivation & Objective

• Assess whether attention weights are uniquely determined by the model outputs (attention identifiability).
• Assess whether input token identities persist in contextual embeddings across layers (token identifiability).
• Quantify how context mixes into hidden embeddings (context contribution) using gradient-based attribution.
• Propose tools to improve interpretability of attention (effective attention).
• Provide empirical evidence on how identity information and context evolve through Transformer layers.

Proposed method

• Theoretically derive an upper bound on the rank of the transformed value matrix T = E W_V H to show non-identifiability when sequence length exceeds the attention head dimension dv.
• Characterize the left null space LN(T) and prove the existence of non-trivial à in LN(T) that satisfy simplex constraints, implying non-uniqueness of attention weights.
• Introduce effective attention A⊥ = A − ProjectionLN(T)(A) to isolate the output-influencing component of attention.
• Empirically validate token identifiability by nearest-neighbor mapping from hidden embeddings to input tokens across layers using cosine and L2 similarities.
• Define Hidden Token Attribution based on gradient attributions to quantify how input tokens contribute to intermediate embeddings.
• Use gradient-based analysis to study how context contributes to hidden embeddings and quantify local vs. distant token influence.

Experimental results

Research questions

• RQ1 Is attention weight distribution identifiable from Transformer outputs for a given input sequence?
• RQ2 Do contextual embeddings preserve identity information of input tokens across layers?
• RQ3 How is context aggregated into hidden embeddings, and to what extent do tokens mix with context across depth?
• RQ4 Can we develop diagnostic tools (effective attention) to better interpret attention beyond raw weights?
• RQ5 How local or global are the context contributions to token embeddings as depth increases?

Key findings

• Attention weights are not identifiable when sequence length exceeds the attention head dimension; there exist infinitely many attention configurations yielding the same output.
• Effective attention diverges from raw attention as sequence length grows, offering a more faithful diagnostic of what influences outputs.
• Input tokens largely retain identity across layers, with recoverable identity information via linear mappings and nearest-neighbor lookups, especially in earlier layers.
• Identity information is mainly encoded in embedding angles and decreases in deeper layers, indicating gradual loss of exact token identity.
• Hidden Token Attribution reveals strong mixing of token and context information in embeddings; context aggregation is predominantly local, with more distant contributions emerging in deeper layers.
• Overall, self-attention distributions are not directly interpretable; the paper provides tools to better understand Transformer internals.