[Paper Review] ERNIE: Enhanced Representation through Knowledge Integration

Yu Sun, Shuohuan Wang+8 more|arXiv (Cornell University)|Apr 19, 2019

Topic Modeling20 references767 citations

TL;DR

ERNIE introduces knowledge masking at entity, phrase, and basic levels to enrich pre-trained language representations, achieving state-of-the-art results on five Chinese NLP tasks compared to BERT.

ABSTRACT

We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the masking strategy of BERT, ERNIE is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking. Entity-level strategy masks entities which are usually composed of multiple words.Phrase-level strategy masks the whole phrase which is composed of several words standing together as a conceptual unit.Experimental results show that ERNIE outperforms other baseline methods, achieving new state-of-the-art results on five Chinese natural language processing tasks including natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering. We also demonstrate that ERNIE has more powerful knowledge inference capacity on a cloze test.

Motivation & Objective

• Motivate improving language representations by integrating prior knowledge into pre-training.
• Develop a masking-based pre-training paradigm that captures entity and phrase-level information.
• Demonstrate gains on Chinese NLP tasks by pre-training on heterogeneous data.

Proposed method

• Use Transformer encoder as in BERT for basic representation learning.
• Apply three-stage knowledge masking: basic-level, phrase-level, and entity-level masking.
• Introduce DLM (Dialogue Language Model) to model query-response dialogue structure.
• Pre-train on heterogeneous Chinese data (Wikepedia, Baike, Baidu News, Baidu Tieba) and incorporate dialogue signals.
• Evaluate on five Chinese NLP tasks to assess knowledge integration benefits.

Experimental results

Research questions

• RQ1 Does masking knowledge at phrase and entity levels improve language representations beyond word-level masking?
• RQ2 What is the contribution of the Dialogue Language Model (DLM) task to downstream performance?
• RQ3 How does ERNIE perform relative to BERT when pre-trained on heterogeneous Chinese data?
• RQ4 Can knowledge-informed pre-training transfer to multiple Chinese NLP tasks effectively?

Key findings

• ERNIE outperforms BERT on all five Chinese NLP tasks studied.
• XNLI, MSRA-NER, ChnSentiCorp, and NLPCC-DBQA show more than 1% absolute accuracy improvement over BERT.
• Phrase-level and entity-level masking provide additional gains beyond basic masking.
• DLM contributes 0.7–1.0% development/test accuracy improvements on XNLI.
• Knowledge masking strategies plus heterogeneous pre-training yield stronger language representations.