[Paper Review] ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

Yu Sun, Shuohuan Wang+5 more|arXiv (Cornell University)|Jul 29, 2019

Topic Modeling30 references74 citations

TL;DR

ERNIE 2.0 extends continual pre-training to incrementally learn lexical, syntactic, and semantic knowledge, achieving improvements over BERT and XLNet on GLUE English tasks and Chinese datasets.

ABSTRACT

Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.

Motivation & Objective

• Motivate the need to go beyond simple co-occurrence in pre-training by leveraging lexical, syntactic, and semantic information in corpora.
• Propose a continual multi-task pre-training framework (ERNIE 2.0) to incrementally construct and learn diverse pre-training tasks.
• Demonstrate improvements over BERT and XLNet on English GLUE benchmarks and various Chinese NLP tasks.

Proposed method

• Construct word-aware, structure-aware, and semantic-aware pre-training tasks using self-supervised or weak-supervised signals from large corpora.
• Employ a shared Transformer encoder with a task embedding to enable knowledge transfer across tasks.
• Implement continual multi-task learning to update the model with new tasks while preserving previously learned knowledge, using N training iterations per task to balance efficiency and forgetting.
• Use a [CLS] token and [SEP] separators within a Transformer framework, with task embeddings to distinguish tasks.
• Fine-tune the pre-trained ERNIE 2.0 model on downstream tasks such as QA, NLI, and semantic similarity.

Experimental results

Research questions

• RQ1 Can continual multi-task pre-training leveraging lexical, syntactic, and semantic signals yield better language representations than single-task pre-training?
• RQ2 How can we train multiple pre-training tasks in a continual fashion without forgetting previously learned knowledge while remaining efficient?
• RQ3 Do ERNIE 2.0 representations achieve superior performance on standard English benchmarks (GLUE) and Chinese NLP tasks compared to BERT and XLNet?

Key findings

• ERNIE 2.0 outperforms BERT and XLNet on 16 tasks, including English GLUE benchmarks and several Chinese tasks.
• The English GLUE results show ERNIE 2.0 LARGE surpassing BERT LARGE and XLNet LARGE on most tasks, achieving a GLUE test score of 83.6 and a 3.1% improvement over the prior state-of-the-art BERT LARGE.
• On Chinese tasks, ERNIE 2.0 LARGE yields the best performance across nine tasks, with ERNIE 1.0 BASE already outperforming BERT on some tasks, and ERNIE 2.0 further advancing results.
• A continual multi-task learning strategy outperforms both multi-task learning from scratch and traditional continual learning, demonstrating effective knowledge retention and task adaptation.