[Paper Review] DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

Pengcheng He, Jianfeng Gao+1 more|arXiv (Cornell University)|Nov 18, 2021

Topic Modeling48 references394 citations

TL;DR

DeBERTaV3 replaces MLM with RTD and introduces gradient-disentangled embedding sharing (GDES) to combine ELECTRA-style pre-training with DeBERTa’s disentangled attention, achieving SOTA results for similar-model-structure PLMs in GLUE, SQuAD v2.0, and cross-lingual XNLI.

ABSTRACT

This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the "tug-of-war" dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mDeBERTa and observed a larger improvement over strong baselines compared to English models. For example, the mDeBERTa Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. We have made our pre-trained models and inference code publicly available at https://github.com/microsoft/DeBERTa.

Motivation & Objective

• Investigate improving pre-training efficiency and downstream performance by combining RTD with DeBERTa.
• Identify and address embedding-sharing conflicts that hinder training in ELECTRA-style setups.
• Propose a gradient-disentangled embedding sharing (GDES) strategy to balance generator and discriminator objectives.
• Evaluate DeBERTaV3 variants on GLUE, SQuAD v2.0, and multilingual XNLI benchmarks to establish new SOTA under similar model scales.

Proposed method

• Replace MLM in DeBERTa with RTD to form DeBERTaV3.
• Analyze embedding-sharing dynamics and demonstrate drawbacks of standard ES (embedding sharing) and NES (no sharing).
• Propose Gradient-Disentangled Embedding Sharing (GDES) where E_G is shared with E_D but gradients from RTD do not flow to the generator embeddings via a stop-gradient mechanism, implemented as E_D = sg(E_G) + E_Delta.
• Re-parameterize discriminator embeddings as E_D = sg(E_G) + E_Delta with E_Delta initialized to zero and updated only through RTD, while E_G is updated via MLM.
• Pre-train multiple model sizes (Large, Base, Small, XSmall) with standard DeBERTa settings, using 160GB of data, 500k steps, batch size 8192, and AdamW optimizer.
• Evaluate on GLUE (eight tasks), MNLI, SQuAD v2.0, RACE, ReCoRD, SWAG, CoNLL-2003, and XNLI across English and multilingual setups.

Experimental results

Research questions

• RQ1 Does replacing MLM with RTD improve pre-training efficiency and downstream NLU performance for DeBERTa?
• RQ2 How does embedding sharing between generator and discriminator affect training dynamics and model quality?
• RQ3 Can a gradient-disentangled sharing strategy (GDES) retain embedding-sharing benefits while avoiding conflicting gradients?
• RQ4 What are the gains of DeBERTaV3 and mDeBERTaV3 on GLUE, MNLI, SQuAD v2.0, and XNLI compared to baselines?

Key findings

• RTD-based DeBERTaV3 significantly outperforms MLM-based DeBERTa on MNLI and SQuAD v2.0 in initial comparisons.
• Embedding sharing in ELECTRA (ES) can slow training due to tug-of-war between generator and discriminator objectives; NES avoids gradient interference but provides limited downstream gains.
• GDES enables generator-discriminator gradient disentanglement while sharing embeddings, yielding faster convergence and higher downstream performance than ES and NES.
• DeBERTaV3 Large achieves a GLUE average of 91.37%, surpassing DeBERTa and ELECTRA baselines by 1.37 and 1.91 points respectively.
• mDeBERTaV3 Base achieves 79.8% average XNLI accuracy, outperforming XLM-R Base and mT5 Base by 3.6 and 4.4 percentage points respectively, establishing a new SOTA for multilingual models with similar size.