[Paper Review] DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Pengcheng He, Xiaodong Liu+2 more|arXiv (Cornell University)|Jun 5, 2020

Topic Modeling61 references422 citations

TL;DR

DeBERTa introduces disentangled attention with separate content and position vectors and an enhanced mask decoder, plus scale-invariant fine-tuning, to outperform prior PLMs on NLU and NLG tasks, including surpassing human performance on SuperGLUE with a 1.5B parameter model.

ABSTRACT

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, out performing the human baseline by a decent margin (90.3 versus 89.8).

Motivation & Objective

• Improve pre-training efficiency and downstream NLP performance over BERT/RoBERTa baselines.
• Introduce a disentangled attention mechanism that separates content and position information.
• Incorporate absolute position information in decoding (Enhanced Mask Decoder) to aid MLM tasks.
• Apply virtual adversarial training for robust fine-tuning (SiFT) to improve generalization.

Proposed method

• Represent each token with two vectors: one for content, one for position.
• Compute attention using four components: content-to-content, content-to-position, position-to-content, (and typically omit) position-to-position.
• Use a relative position embedding with a fixed 2k range for efficiency.
• Incorporate absolute position information after Transformer layers via Enhanced Mask Decoder during MLM pre-training.
• Introduce Scale-invariant Fine-Tuning (SiFT) for robust downstream fine-tuning by perturbing normalized embeddings.
• Pre-train large (1.5B) and base DeBERTa models on ~78–160G text datasets and evaluate on GLUE/SuperGLUE/NLG tasks.

Experimental results

Research questions

• RQ1 Does disentangled attention improve performance over standard self-attention on NLP tasks?
• RQ2 What is the impact of incorporating absolute positions via Enhanced Mask Decoder on MLM pre-training?
• RQ3 Can SiFT improve fine-tuning robustness and generalization for large DeBERTa models?
• RQ4 How does DeBERTa scale in performance with model size compared to peers like RoBERTa, XLNet, ELECTRA?
• RQ5 Is DeBERTa able to surpass human baselines on challenging benchmarks like SuperGLUE?

Key findings

• DeBERTa large outperforms RoBERTa large and XLNet large on most GLUE tasks with similar training data, achieving higher average scores.
• DeBERTa large yields MNLI and SQuAD gains (e.g., MNLI +0.9%, SQuAD v2.0 +2.3%, RACE +3.6%) compared to RoBERTa-Large.
• A 1.5B DeBERTa model achieves 89.9 on SuperGLUE macro-average, surpassing the human baseline (89.8).
• The DeBERTa base model (12L, 768 hidden) consistently outperforms RoBERTa and XLNet on MNLI, SQuAD, and RACE in ablations.
• Ablation shows removing EMD or either C2P or P2C terms degrades performance across benchmarks, confirming each component’s contribution.
• Scale-up with 1.5B parameters improves both NLU and NLG tasks and offers a more energy-efficient alternative to comparably large models (e.g., T5 11B).
• On SuperGLUE, DeBERTa 1.5B plus SiFT achieves competitive scores and the ensemble sets top rankings as of 2021.