[Paper Review] CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model

Liang Xu, Xuanwei Zhang+1 more|arXiv (Cornell University)|Mar 3, 2020

Topic ModelingComputer Science11 references34 citations

TL;DR

Introduces CLUECorpus2020, a 100 GB large-scale Chinese raw corpus for pre-training language models, along with a compact vocab and pre-trained models; demonstrates strong performance on Chinese NLP benchmarks and efficiency gains.

ABSTRACT

In this paper, we introduce the Chinese corpus from CLUE organization, CLUECorpus2020, a large-scale corpus that can be used directly for self-supervised learning such as pre-training of a language model, or language generation. It has 100G raw corpus with 35 billion Chinese characters, which is retrieved from Common Crawl. To better understand this corpus, we conduct language understanding experiments on both small and large scale, and results show that the models trained on this corpus can achieve excellent performance on Chinese. We release a new Chinese vocabulary with a size of 8K, which is only one-third of the vocabulary size used in Chinese Bert released by Google. It saves computational cost and memory while works as good as original vocabulary. We also release both large and tiny versions of the pre-trained model on this corpus. The former achieves the state-of-the-art result, and the latter retains most precision while accelerating training and prediction speed for eight times compared to Bert-base. To facilitate future work on self-supervised learning on Chinese, we release our dataset, new vocabulary, codes, and pre-trained models on Github.

Motivation & Objective

• Provide a large-scale, high-quality Chinese unlabeled corpus suitable for pre-training language models and generation.
• Evaluate how training on CLUECorpus2020 impacts Chinese NLP task performance across varying data sizes and vocabularies.
• Propose and release a compact Chinese vocabulary (vocab_clue) and enable both large and tiny pre-trained models for efficient training and inference.
• Demonstrate potential benefits of transfer learning across related Chinese NLP tasks using pre-trained models.

Proposed method

• Assemble a 100 GB Chinese corpus from Common Crawl with filtering rules tailored for Chinese text quality and deduplication.
• Define a pre-training-ready format (one sentence per line, empty line per document) and select a train/dev/test split (roughly 99:0.5:0.5).
• Create a compact vocabulary (vocab_clue) by removing redundant tokens and tailoring Chinese/English tokenization, resulting in 8K size.
• Train multiple BERT-style models on CLUECorpus2020 and compare with Google’s vocab and Wiki/C5 baselines using the CLUE benchmark tasks.
• Experiment with model variants including attention mechanisms (e.g., a minus-and-multiplication variant) and with large vs tiny pre-trained models to assess efficiency and accuracy.
• Assess the impact of data scale by varying training data (1 GB vs 3 GB vs 100 GB) and steps, and compare performance against baselines.

Experimental results

Research questions

• RQ1 How does pre-training on CLUECorpus2020 influence Chinese language understanding across standard benchmarks?
• RQ2 What are the effects of using the proposed vocab_clue versus Google’s original vocabulary on downstream tasks?
• RQ3 What is the impact of increasing pre-training data size and training steps on model performance?
• RQ4 Can a compact, efficient model (tiny variants) achieve competitive performance with substantial speedups?
• RQ5 Does transfer learning across related Chinese tasks improve performance for sentence-pair tasks?

Key findings

• Models trained on CLUECorpus2020 achieve competitive or state-of-the-art results on Chinese benchmarks when fully trained on the dataset.
• A compact 8K vocabulary (vocab_clue) yields comparable downstream performance to larger vocabularies while reducing parameters and speeding training.
• The CLUE vocabulary enables faster training (approximately 15.4% speedup) with smaller model sizes compared to Google’s vocabulary.
• Increasing training data (from 1 GB to 3 GB or more) generally improves performance across tasks.
• Tiny variants like RoBERTa-tiny-clue retain most precision while substantially increasing inference/training speed (up to about 8x faster than BERT-base in some setups).
• Transfer learning across related tasks (e.g., CMNLI to AFQMC) can boost performance for sentence-pair tasks.