[Paper Review] DeepNet: Scaling Transformers to 1,000 Layers

Hongyu Wang, Shuming Ma+4 more|arXiv (Cornell University)|Mar 1, 2022

Neural Networks and Reservoir Computing54 citations

TL;DR

DEEPNET introduces DEEPNORM, a normalization method at residual connections with initialization to bound model updates, enabling Transformer depth up to 1,000 layers and achieving strong multilingual MT performance.

ABSTRACT

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.

Motivation & Objective

• Motivate and address training instability when scaling Transformers to extreme depths (up to 1,000 layers).
• Propose a simple normalization and initialization scheme (DEEPNORM) to bound model updates and stabilize optimization.
• Demonstrate stability and performance gains across machine translation benchmarks, including multilingual settings.
• Provide practical initialization and architectural guidelines for encoder-only, decoder-only, and encoder-decoder models.

Proposed method

• Define and apply a new normalization at residual connections: xl+1 = LN(α xl + Gl(xl, θl)).
• Scale sub-layer weights inside residual branches by a architecture-dependent factor β.
• Derive initialization schemes to bound ||ΔF||, ensuring updates remain O(η) under SGD/Adam.
• Theoretically bound model updates for N-layer DEEPNET and for encoder-decoder variants (Theorems 4.2 and 4.3).
• Empirically validate on WMT En–De, IWSLT De–En, and massively multilingual OPUS-100 and Flores datasets.
• Provide concrete practical initialization values and scaling rules for encoder-only, decoder-only, and encoder-decoder architectures.

Experimental results

Research questions

• RQ1 Can Transformers be trained stably when scaled to 1,000 layers?
• RQ2 Does DEEPNORM stabilize training while preserving or improving performance compared with Post-LN and Pre-LN baselines?
• RQ3 How do initialization and residual scaling affect the magnitude of model updates and gradient flow in extremely deep Transformers?
• RQ4 What are the practical gains in machine translation performance for both bilingual and multilingual settings when depth is increased with DEEPNORM?

Key findings

• DEEPNET with DEEPNORM achieves stable training up to 1,000 layers (2,500 sublayers) where previous methods diverged.
• On WMT-17 En–De, 200-layer DEEPNET reaches 28.9 BLEU compared with a 48-layer 12B-param baseline, demonstrating strong efficiency in depth.
• In multilingual MT, DEEPNET with 200 layers (3.2B params) outperforms a 48-layer 12B baseline by 4.4 BLEU on OPUS-100 averages (WMT/OPUS/TED/Flores).
• DEEPNET scales to 1,000 layers with 3.8B parameters and achieves BLEU scores of 33.9 (WMT), 30.2 (OPUS) and 18.6 (Flores) in multilingual evaluation, surpassing M2M-100 (12B, 48 layers).
• Theoretical analysis shows DEEPNORM bounds model updates by a constant, mitigating exploding updates and gradient issues in deep Transformers.