[Paper Review] ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Yu Sun, Shuohuan Wang+20 more|arXiv (Cornell University)|Jul 5, 2021

Topic Modeling79 references193 citations

TL;DR

ERNIE 3.0 introduces a large-scale, knowledge-enhanced pre-training framework that fuses auto-regressive and auto-encoding objectives, enabling strong performance on both language understanding and generation tasks, guided by a continual multi-paradigms approach and a 4TB mixture of plain text and knowledge graph data.

ABSTRACT

Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up pre-trained language models can improve their generalization abilities. Particularly, the GPT-3 model with 175 billion parameters shows its strong task-agnostic zero-shot/few-shot learning capabilities. Despite their success, these large-scale models are trained on plain texts without introducing knowledge such as linguistic knowledge and world knowledge. In addition, most large-scale models are trained in an auto-regressive way. As a result, this kind of traditional fine-tuning approach demonstrates relatively weak performance when solving downstream language understanding tasks. In order to solve the above problems, we propose a unified framework named ERNIE 3.0 for pre-training large-scale knowledge enhanced models. It fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning. We trained the model with 10 billion parameters on a 4TB corpus consisting of plain texts and a large-scale knowledge graph. Empirical results show that the model outperforms the state-of-the-art models on 54 Chinese NLP tasks, and its English version achieves the first place on the SuperGLUE benchmark (July 3, 2021), surpassing the human performance by +0.8% (90.6% vs. 89.8%).

Motivation & Objective

• Motivate the integration of knowledge (linguistic and world knowledge) into large-scale pre-training to improve downstream task performance.
• Propose a unified Continual Multi-Paradigms Unified Pre-training Framework that supports NLU, NLG, and knowledge extraction tasks.
• Build a 10B-parameter ERNIE 3.0 model trained on 4TB of plain text plus a knowledge graph.
• Demonstrate improvements over state-of-the-art models across a broad suite of NLP benchmarks (54 Chinese tasks) and English SuperGLUE.
• Provide training strategies (progressive training, multi-task objectives) and data curation methods that enable efficient large-scale knowledge-enhanced pre-training.

Proposed method

• Propose a Universal Representation Module (shared) plus two Task-specific Representation Modules (NLU and NLG) to share underlying features while learning task-specific top-level representations.
• Backbone uses Transformer-XL to model long-range dependencies with an auxiliary memory for the universal module; NLU uses bidirectional encoding while NLG uses unidirectional encoding in task-specific modules.
• Employ a suite of pre-training tasks spanning: word-aware (knowledge masked language modeling), structure-aware (sentence reordering, sentence distance), knowledge-aware (UKTP: universal knowledge-text prediction) and document-level language modeling for generation.
• Incorporate continual multi-paradigms learning to enable multi-task pre-training with shared encoders and task-specific decoders, enabling zero-shot, few-shot, or fine-tuning for downstream tasks.
• Data: a 4TB Chinese corpus combining plain text and a Baidu knowledge graph; deduplication, filtering, and segmentation steps to improve data quality.
• Pre-training settings: 48-layer universal Transformer-XL for the shared module (4,096 hidden, 64 heads) and 12-layer task-specific modules (768 hidden, 12 heads); total 10B parameters; training with 375B tokens on 6144 NVIDIA V100 GPUs using PaddlePaddle; progressive training with memory and sequence-length considerations.

Experimental results

Research questions

• RQ1 Can a knowledge-enhanced pre-training framework improve both language understanding and generation tasks at a large scale?
• RQ2 Does fusing auto-regressive and auto-encoding objectives with continual multi-task learning yield better representations for downstream tasks compared to single-paradigm pre-training?
• RQ3 What is the impact of incorporating a knowledge graph alongside plain text on a 10B-parameter model's performance across diverse NLP benchmarks?
• RQ4 How does incremental, task-specific fine-tuning behave when starting from a shared universal representation and task-specific modules?

Key findings

• ERNIE 3.0 achieves state-of-the-art results across 54 Chinese NLP benchmarks, outperforming strong baselines significantly.
• The English version of ERNIE 3.0 ranks first on the SuperGLUE benchmark at the time of reporting (90.6% vs. 89.8% human parity).
• The unified framework enables zero-shot, few-shot, and fine-tuning paradigms for both NLU and NLG tasks.
• The combination of large-scale knowledge integration with continual multi-paradigms pre-training yields strong improvements over prior models on both understanding and generation tasks.
• Progressive training and a large, high-quality corpus with a knowledge graph contribute to training stability and performance gains.
• The model demonstrates robust performance across a wide range of tasks, including sentiment analysis, natural language inference, reading comprehension, named entity recognition, and generation tasks.