[Paper Review] NEZHA: Neural Contextualized Representation for Chinese Language Understanding

Victor Junqiu Wei, Xiaozhe Ren+8 more|arXiv (Cornell University)|Aug 31, 2019

Topic ModelingComputer Science18 references86 citations

TL;DR

NEZHA pre-trains Chinese language models on large corpora using functional relative positional encoding, whole word masking, mixed precision, and LAMB optimizer, achieving strong results on Chinese NLU tasks when fine-tuned.

ABSTRACT

The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora. In this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed representation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks. The current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional Relative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy, Mixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA achieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including named entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti) and natural language inference (XNLI).

Motivation & Objective

• Study effective pre-training strategies for Chinese language models beyond BERT.
• Evaluate positional encoding, masking, data sources, and sequence length on downstream tasks.
• Demonstrate state-of-the-art performance on multiple Chinese NLU benchmarks through fine-tuning.

Proposed method

• Adopt BERT-like Transformer encoder with functional relative positional encoding (FRPE) as the core positional method.
• Use Whole Word Masking (WWM) for Chinese word-level masking to improve pre-training signals.
• Apply mixed-precision training to accelerate training and reduce memory usage.
• Utilize the LAMB optimizer to enable large-batch training and stable convergence.
• Train NEZHA models from scratch on large Chinese corpora (Wikipedia, Baike, News) and compare with BERT and ERNIE-based models on downstream tasks.

Experimental results

Research questions

• RQ1 Does functional relative positional encoding (FRPE) provide consistent gains over absolute or other relative encodings for Chinese pre-trained models?
• RQ2 What is the impact of Whole Word Masking on Chinese pre-trained models compared with standard masking?
• RQ3 How do training data sources and sequence lengths affect pre-training effectiveness and downstream performance?
• RQ4 Can NEZHA achieve state-of-the-art results on typical Chinese NLU tasks after fine-tuning?
• RQ5 What is the practical benefit of mixed precision training and LAMB optimization for large-scale pre-training?

Key findings

• FRPE consistently improves performance over PAPE and PRPE on several tasks, notably helping longer-context tasks like CMRC.
• WWM provides measurable gains over standard masking in multiple Chinese NLU benchmarks.
• Longer training sequences and diverse data sources (Wikipedia, Baike, News) contribute to better downstream results.
• NEZHA-base and NEZHA-large variants achieve state-of-the-art or strong results across tasks such as CMRC, XNLI, LCQMC, PD-NER, and ChnSenti when fine-tuned.
• In ablations, FRPE shows notable advantages, and the combination of WWM, longer sequences, and mixed-precision with LAMB yields favorable performance and training efficiency.