[论文解读] ResiDual: Transformer with Dual Residual Connections

Shufang Xie, Huishuai Zhang+7 more|arXiv (Cornell University)|Apr 28, 2023

Natural Language Processing Techniques被引用 6

一句话总结

该论文提出ResiDual，一种新型Transformer架构，采用双残差连接，结合预层归一化（Pre-LN）与后层归一化（Post-LN），以克服梯度消失与表征崩溃问题。通过融合两种残差路径，ResiDual实现了更优的训练稳定性和模型容量，在机器翻译基准测试中表现优于Post-LN与Pre-LN变体，深层模型的性能提升最高达1.1 BLEU点。

ABSTRACT

Transformer networks have become the preferred architecture for many tasks due to their state-of-the-art performance. However, the optimal way to implement residual connections in Transformer, which are essential for effective training, is still debated. Two widely used variants are the Post-Layer-Normalization (Post-LN) and Pre-Layer-Normalization (Pre-LN) Transformers, which apply layer normalization after each residual block's output or before each residual block's input, respectively. While both variants enjoy their advantages, they also suffer from severe limitations: Post-LN causes gradient vanishing issue that hinders training deep Transformers, and Pre-LN causes representation collapse issue that limits model capacity. In this paper, we propose ResiDual, a novel Transformer architecture with Pre-Post-LN (PPLN), which fuses the connections in Post-LN and Pre-LN together and inherits their advantages while avoids their limitations. We conduct both theoretical analyses and empirical experiments to verify the effectiveness of ResiDual. Theoretically, we prove that ResiDual has a lower bound on the gradient to avoid the vanishing issue due to the residual connection from Pre-LN. Moreover, ResiDual also has diverse model representations to avoid the collapse issue due to the residual connection from Post-LN. Empirically, ResiDual outperforms both Post-LN and Pre-LN on several machine translation benchmarks across different network depths and data sizes. Thanks to the good theoretical and empirical performance, ResiDual Transformer can serve as a foundation architecture for different AI models (e.g., large language models). Our code is available at https://github.com/microsoft/ResiDual.

研究动机与目标

• 为解决现有Transformer残差连接变体的局限性，特别是Post-LN中的梯度消失问题与Pre-LN中的表征崩溃问题。
• 设计一种统一的Transformer架构，兼具Post-LN与Pre-LN的优势，同时避免其各自缺陷。
• 在不依赖学习率预热的情况下，实现深层Transformer模型的稳定且高效训练。
• 通过架构创新提升深层网络的模型容量与表征多样性。

提出的方法

• 提出一种名为ResiDual的新架构，集成两种残差连接：一种受Pre-LN启发以稳定梯度，另一种受Post-LN启发以保持表征多样性。
• 引入预-后层归一化（PPLN），在每个残差块之前和之后均应用层归一化。
• 采用双残差路径：一条残差跳跃连接从输入到模块输出（Pre-LN风格），另一条从输入到残差输出（Post-LN风格）。
• 理论上证明ResiDual具有有界梯度范数，可避免梯度消失，并保持多样化的隐藏表征，防止表征崩溃。
• 通过理论分析表明，双残差机制可确保稳定的梯度流动与丰富的表征学习。
• 在多个机器翻译基准（包括WMT与OPUS-100）上进行实证验证，覆盖不同深度与数据规模。

实验结果

研究问题

• RQ1 能否设计一种Transformer架构，同时避免深层网络中的梯度消失与表征崩溃？
• RQ2 将Pre-LN与Post-LN残差连接结合，是否能带来优于单一变体的训练稳定性和模型性能？
• RQ3 所提出的双残差机制能否减少或消除训练过程中对学习率预热的依赖？
• RQ4 与Post-LN和Pre-LN等现有变体相比，该模型在不同网络深度与数据规模下的表现如何？
• RQ5 双残差设计在多大程度上增强了深层Transformer的表征多样性与模型容量？

主要发现

• ResiDual在所有评估基准上均优于Post-LN与Pre-LN，在E12D12设置下相较标准Pre-LN实现0.9 BLEU点的性能提升。
• 在WMT数据集上，ResiDual在E18D18模型上达到27.65 BLEU，较Pre-LN基线提升1.1点。
• 在OPUS-100数据集上，ResiDual在仅18层的情况下实现ALL分数31.0 BLEU，与100层的DeepNet模型性能相当。
• ResiDual在无学习率预热条件下成功训练，而Post-LN在无预热时无法收敛，表明其训练稳定性显著提升。
• 理论分析证实，ResiDual因梯度范数有界而避免梯度消失，并保持与Post-LN相当的表征多样性。
• 实证结果表明，ResiDual在模型深度增加时仍保持强劲性能，表明其具备更强的泛化能力与容量利用效率。