[Paper Review] Transformers without Tears: Improving the Normalization of Self-Attention

Toan Nguyen, Julián Salazar|arXiv (Cornell University)|Oct 14, 2019

Topic Modeling37 references130 citations

TL;DR

The paper introduces normalization-centric changes (PreNorm, ScaleNorm, FixNorm) to Transformer training, showing improved low-resource NMT performance and competitive high-resource results, with faster training and smoother gradient behavior.

ABSTRACT

We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PreNorm) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose $\ell_2$ normalization with a single scale parameter (ScaleNorm) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FixNorm). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT'15 English-Vietnamese. We observe sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT'14 English-German), ScaleNorm and FixNorm remain competitive but PreNorm degrades performance.

Motivation & Objective

• Investigate how normalization placement and types affect Transformer convergence and performance in low-resource NMT.
• Assess whether replacing LayerNorm with ScaleNorm improves training speed and accuracy.
• Evaluate the impact of fixing word embedding norms (FixNorm) and scaling embeddings (ScaleNorm) on translation quality.
• Demonstrate whether these changes yield gains across multiple language pairs with varying resource levels.

Proposed method

• Compare PostNorm and PreNorm residual configurations across multiple low-resource NMT tasks.
• Replace LayerNorm with ScaleNorm and incorporate FixNorm to study training dynamics and performance.
• Experiment with learning-rate schedules including warmup-free training and validation-based decay to assess stability.
• Share preprocessing and training scripts and evaluate BLEU scores on TED/IWSLT-based corpora.

Experimental results

Research questions

• RQ1 Does PreNorm improve training stability and performance in low-resource Transformer NMT compared to PostNorm?
• RQ2 Do ScaleNorm and FixNorm provide faster training and higher BLEU across low-resource language pairs?
• RQ3 Can ScaleNorm with FixNorm maintain competitive performance in high-resource settings like WMT14 English-German?
• RQ4 How do the proposed normalization changes affect gradient norms and activation scaling across decoder depth?

Key findings

• PreNorm enables warmup-free training with large learning rates in low-resource settings, improving stability and performance.
• ScaleNorm reduces computational cost and can improve BLEU on several low-resource pairs, often outperforming LayerNorm.
• FixNorm consistently contributes additional improvements, especially when combined with ScaleNorm.
• The final combination PreNorm + FixNorm + ScaleNorm yields an average +1.1 BLEU across five low-resource pairs, with notable gains on IWSLT '15 English-Vietnamese to 32.8 BLEU.
• ScaleNorm and FixNorm often match or exceed LayerNorm performance in high-resource WMT14 English-German, with ScaleNorm offering speedups.
• PreNorm shows improved gradient stability (smoother norms) compared to PostNorm during training.