[Paper Review] Attention is Not Only a Weight: Analyzing Transformers with Vector Norms

Goro Kobayashi, Tatsuki Kuribayashi+2 more|arXiv (Cornell University)|Apr 21, 2020

Topic Modeling36 references17 citations

TL;DR

This paper proposes norm-based analysis to augment traditional weight-based attention analysis in Transformers by incorporating the L2 norms of transformed input vectors, revealing that attention outputs depend on both attention weights and vector magnitudes. The method uncovers that BERT underweights special tokens and controls frequent word contributions via norm modulation, while also enabling more accurate word alignment in neural machine translation systems than weight-based methods alone.

ABSTRACT

Attention is a key component of Transformers, which have recently achieved considerable success in natural language processing. Hence, attention is being extensively studied to investigate various linguistic capabilities of Transformers, focusing on analyzing the parallels between attention weights and specific linguistic phenomena. This paper shows that attention weights alone are only one of the two factors that determine the output of attention and proposes a norm-based analysis that incorporates the second factor, the norm of the transformed input vectors. The findings of our norm-based analyses of BERT and a Transformer-based neural machine translation system include the following: (i) contrary to previous studies, BERT pays poor attention to special tokens, and (ii) reasonable word alignment can be extracted from attention mechanisms of Transformer. These findings provide insights into the inner workings of Transformers.

Motivation & Objective

• To address the limitation of weight-based attention analysis, which ignores the magnitude of transformed input vectors.
• To propose a norm-based analysis method that accounts for both attention weights and vector norms in attention mechanism outputs.
• To provide deeper insights into how Transformers process linguistic elements like special tokens and frequent words.
• To demonstrate that more accurate word alignments can be extracted from source-target attention in Transformer-based NMT systems by considering vector norms.

Proposed method

• Propose a norm-based analysis framework that evaluates the L2 norm of value vectors after transformation by the projection matrices in the attention mechanism.
• Measure the norm of each input vector after applying the value transformation $\boldsymbol{v}(\boldsymbol{x}_j) = \boldsymbol{x}_j \boldsymbol{W}^V + \boldsymbol{b}^V$ to assess its contribution magnitude.
• Integrate vector norms with attention weights to compute a combined contribution metric, revealing that high weights do not always imply high effective contribution.
• Apply the method to BERT and a Transformer-based NMT system to analyze token-level attention dynamics and word alignment quality.
• Compare norm-based results with standard weight-based analysis to identify discrepancies in perceived attention behavior.
• Use publicly available code to reproduce and validate findings across multiple models and tasks.

Experimental results

Research questions

• RQ1 How do vector norms of transformed input vectors affect the final output of the attention mechanism in Transformers?
• RQ2 Why do previous weight-based analyses fail to accurately represent the true contribution of special tokens like [SEP] in BERT?
• RQ3 Can norm-based analysis improve the quality of word alignment extraction in Transformer-based neural machine translation systems?
• RQ4 To what extent do frequent, low-information words influence attention outputs, and how is this controlled?
• RQ5 Can the attention mechanism be better interpreted by considering both attention weights and the magnitude of transformed vectors?

Key findings

• BERT's attention mechanism pays significantly less effective attention to special tokens such as [SEP] than indicated by attention weights alone, due to low vector norms.
• The contribution of frequent, low-information words to attention outputs is actively suppressed by the model through norm modulation, not just attention weighting.
• Norm-based analysis reveals that attention weights alone are insufficient for interpreting attention dynamics, as high weights can be canceled by small vector norms.
• In Transformer-based NMT systems, word alignments extracted using norm-based attention are more accurate than those derived from attention weights alone.
• The proposed norm-based method provides a more nuanced and accurate interpretation of attention behavior than weight-based analysis, especially in identifying true information contributors.
• The method is robust across different input lengths and does not suffer from the limitations of prior approaches that only consider binary cancellation of attention weights.