[Paper Review] Very Deep Transformers for Neural Machine Translation

Xiaodong Liu, Kevin Duh+2 more|arXiv (Cornell University)|Aug 18, 2020

Natural Language Processing Techniques35 references71 citations

TL;DR

The paper demonstrates that standard Transformers can be trained very deeply (up to 60 encoder layers and 12 decoder layers) using ADMIN initialization, achieving up to 2.5 BLEU gains and new state-of-the-art results on WMT’14 EN-FR and EN-DE with and without back-translation.

ABSTRACT

We explore the application of very deep Transformer models for Neural Machine Translation (NMT). Using a simple yet effective initialization technique that stabilizes training, we show that it is feasible to build standard Transformer-based models with up to 60 encoder layers and 12 decoder layers. These deep models outperform their baseline 6-layer counterparts by as much as 2.5 BLEU, and achieve new state-of-the-art benchmark results on WMT14 English-French (43.8 BLEU and 46.4 BLEU with back-translation) and WMT14 English-German (30.1 BLEU).The code and trained models will be publicly available at: https://github.com/namisan/exdeep-nmt.

Motivation & Objective

• Investigate whether deeper Transformer architectures can improve NMT performance.
• Develop a stable training initialization to enable very deep models without architectural changes.
• Evaluate depth effects on standard WMT’14 EN-FR and EN-DE benchmarks (with and without back-translation).
• Provide reproducible results and release code/models to facilitate further research.

Proposed method

• Apply ADMIN initialization to balance residual and feed-forward paths in deep Transformer blocks.
• Use a profiling phase to estimate per-layer residual variance and set layer-scaling factors ω_i.
• Train standard post-LN Transformer architectures with up to 60 encoder layers and 12 decoder layers under single-GPU-friendly constraints.
• Compare deep ADMIN models to standard 6L-6L baselines on BLEU, TER, and METEOR metrics.
• Conduct ablation studies across encoder/decoder depth and network width.
• Evaluate back-translation effects with deep models on WMT’14 EN-FR.

Experimental results

Research questions

• RQ1 Can standard Transformer architectures be trained effectively at very large depths using a principled initialization?
• RQ2 Does depth provide consistent gains in NMT quality across language pairs (EN-FR, EN-DE) and evaluation metrics?
• RQ3 What is the impact of depth on learning dynamics, such as training perplexity and gradient behavior?
• RQ4 Do deeper models improve translation of low-frequency words and long sentences?
• RQ5 Do back-translation data continue to benefit very deep models similarly to shallower ones?

Key findings

• 60L encoder with 12L decoder (60L-12L ADMIN) achieves up to 2.5 BLEU gains over 6L-6L baselines on EN-FR and EN-DE.
• 60L-12L ADMIN yields BLEU 43.8 (FR) and 30.1 (DE) versus 6L-6L defaults, with statistically significant improvements (p<0.05).
• 60L-12L ADMIN also shows TER and METEOR improvements (e.g., FR TER 40.3 vs 42.2; METEOR 62.4 vs 60.5).
• Pre-LN variants train but underperform post-LN ADMIN deep models; deep post-LN ADMIN models achieve state-of-the-art on both benchmarks.
• Ablation shows deeper encoders are more beneficial than deeper decoders for the same total depth; 60L-12L, 48L-12L, and 36L-36L are statistically tied for best BLEU.
• Back-translation with ADMIN models further improves EN-FR BLEU, achieving 46.4 (36L-12L-768D ADMIN + BT) and 46.0 (60L-12L ADMIN + BT).