[论文解读] DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

Pengcheng He, Jianfeng Gao+1 more|arXiv (Cornell University)|Nov 18, 2021

Topic Modeling参考文献 48被引用 394

一句话总结

DeBERTaV3 用 RTD 取代 MLM，并引入梯度解耦嵌入共享（GDES），将 ELECTRA 风格的预训练与 DeBERTa 的解耦注意力结合起来，在 GLUE、SQuAD v2.0 和跨语言 XNLI 的相似模型结构的 PLMs 中实现 SOTA 结果。

ABSTRACT

This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the "tug-of-war" dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mDeBERTa and observed a larger improvement over strong baselines compared to English models. For example, the mDeBERTa Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. We have made our pre-trained models and inference code publicly available at https://github.com/microsoft/DeBERTa.

研究动机与目标

• 研究通过将 RTD 与 DeBERTa 结合来提升预训练效率和下游性能。
• 识别并解决 ELECTRA 风格设置中影响训练的嵌入共享冲突。
• 提出梯度解耦嵌入共享（GDES）策略，以平衡生成器与判别器的目标。
• 在 GLUE、SQuAD v2.0 及多语言 XNLI 基准上评估 DeBERTaV3 的变体，在相近模型规模下建立新的 SOTA。

提出的方法

• 在 DeBERTa 中用 RTD 替代 MLM，形成 DeBERTaV3。
• 分析嵌入共享的动态，并证明标准 ES（嵌入共享）和 NES（不共享）的弊端。
• 提出梯度解耦嵌入共享（GDES），其中 E_G 与 E_D 共享，但来自 RTD 的梯度通过 stop-gradient 机制不作用于生成器嵌入，形式实现为 E_D = sg(E_G) + E_Delta。
• 将判别器嵌入重新参数化为 E_D = sg(E_G) + E_Delta，其中 E_Delta 初始为零仅通过 RTD 更新，而 E_G 通过 MLM 更新。
• 使用标准 DeBERTa 设置预训练多个模型尺寸（Large, Base, Small, XSmall），使用 160GB 的数据、500k 步、批量大小 8192，以及 AdamW 优化器。
• 在 GLUE（八个任务）、MNLI、SQuAD v2.0、RACE、ReCoRD、SWAG、CoNLL-2003，以及 XNLI（英语与多语言设置）上进行评估。

实验结果

研究问题

• RQ1 将 MLM 替换为 RTD 是否提高了 DeBERTa 的预训练效率和下游 NLU 性能？
• RQ2 生成器与判别器之间的嵌入共享如何影响训练动态和模型质量？
• RQ3 梯度解耦共享策略（GDES）是否能够在避免梯度冲突的同时保留嵌入共享的好处？
• RQ4 相较基线，DeBERTaV3 和 mDeBERTaV3 在 GLUE、MNLI、SQuAD v2.0 和 XNLI 上的提升是多少？

主要发现

• RTD-based DeBERTaV3 significantly outperforms MLM-based DeBERTa on MNLI and SQuAD v2.0 in initial comparisons.
• Embedding sharing in ELECTRA (ES) can slow training due to tug-of-war between generator and discriminator objectives; NES avoids gradient interference but provides limited downstream gains.
• GDES enables generator-discriminator gradient disentanglement while sharing embeddings, yielding faster convergence and higher downstream performance than ES and NES.
• DeBERTaV3 Large achieves a GLUE average of 91.37%, surpassing DeBERTa and ELECTRA baselines by 1.37 and 1.91 points respectively.
• mDeBERTaV3 Base achieves 79.8% average XNLI accuracy, outperforming XLM-R Base and mT5 Base by 3.6 and 4.4 percentage points respectively, establishing a new SOTA for multilingual models with similar size.