[Paper Review] CLUE: A Chinese Language Understanding Evaluation Benchmark

Liang Xu, Hai Hu+30 more|arXiv (Cornell University)|Apr 13, 2020

Topic ModelingComputer Science38 references63 citations

TL;DR

CLUE introduces a large-scale Chinese NLU benchmark with 9 tasks, a 214 GB pre-training corpus, a diagnostic dataset, and a public leaderboard with PyCLUE toolkit to evaluate Chinese models.

ABSTRACT

The advent of natural language understanding (NLU) benchmarks for English, such as GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of tasks. These comprehensive benchmarks have facilitated a broad range of research and applications in natural language processing (NLP). The problem, however, is that most such benchmarks are limited to English, which has made it difficult to replicate many of the successes in English NLU for other languages. To help remedy this issue, we introduce the first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an open-ended, community-driven project that brings together 9 tasks spanning several well-established single-sentence/sentence-pair classification tasks, as well as machine reading comprehension, all on original Chinese text. To establish results on these tasks, we report scores using an exhaustive set of current state-of-the-art pre-trained Chinese models (9 in total). We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on Chinese NLU. Our benchmark is released at https://www.CLUEbenchmarks.com

Motivation & Objective

• Provide a comprehensive, standardized benchmark for Chinese natural language understanding comparable to English benchmarks like GLUE/SuperGLUE.
• Curate diverse Chinese NLU tasks spanning single-sentence, sentence-pair, and machine reading comprehension to probe different model capabilities.
• Release a large-scale, open Chinese pre-training corpus to enable meaningful cross-model comparisons.
• Offer diagnostic data to analyze models’ grasp of Chinese-specific linguistic phenomena and provide tooling for easy adoption and reproduction.

Proposed method

• Assemble nine Chinese NLU tasks (single-sentence, sentence-pair, and MRC) with clear train/dev/test splits.
• Create a large-scale, 214 GB Chinese pre-training corpus totaling ~76 billion words from multiple sources (CLUECorpus2020-small, CLUECorpus2020, CLUEOSCAR).
• Design a linguistically motivated diagnostic dataset crafted by Chinese linguists to probe phenomena like anaphora, monotonicity, and Chinese-specific aspect markers.
• Provide a public online leaderboard with an auto-evaluation system and certify open-source models for reproducibility.
• Develop PyCLUE, a TensorFlow-based toolkit to facilitate training and evaluation across CLUE tasks on standard architectures.

Experimental results

Research questions

• RQ1 How do current pre-trained Chinese language models perform across a diverse set of Chinese NLU tasks?
• RQ2 What is the impact of model size and pre-training data on Chinese NLU performance within CLUE?
• RQ3 To what extent do models struggle with Chinese-specific linguistic phenomena captured by the diagnostic dataset?
• RQ4 How close are model performances to human performance on core CLUE tasks and where is the gap largest?

Key findings

• Larger models and those trained with more pre-training data achieve higher averaged CLUE scores.
• RoBERTa-wwm-ext-large and ALBERT-xxlarge yield the best overall performance among baselines, with notable strength on machine reading tasks.
• Small models (e.g., ALBERT-tiny) show minimal gains on single-sentence/ sentence-pair tasks but substantial performance drop on machine reading tasks, highlighting limits for compact models on global understanding.
• All models show a sizable gap to human performance on several tasks, especially those requiring deep reasoning like WSC-style challenges and free-form MRC (C 3).
• The diagnostic dataset reveals that even strong models struggle with Chinese-specific phenomena, such as monotonicity and lexical semantics distinctions.