[Paper Review] PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

Wei Zeng, Xiaozhe Ren+36 more|arXiv (Cornell University)|Apr 26, 2021

Topic Modeling39 references94 citations

TL;DR

PanGu-α trains up to 200B-parameter Chinese autoregressive language models with five-dimensional auto-parallelism on 2048 Ascend 910 processors, using a 1.1TB high-quality Chinese corpus, and demonstrates few-shot/zero-shot capabilities across Chinese NLP tasks.

ABSTRACT

Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural language understanding and generation with extit{few-shot in-context} learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu-$α$, with up to 200 billion parameters. PanGu-$α$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is implemented based on MindSpore Auto-parallel, which composes five parallelism dimensions to scale the training task to 2048 processors efficiently, including data parallelism, op-level model parallelism, pipeline model parallelism, optimizer model parallelism and rematerialization. To enhance the generalization ability of PanGu-$α$, we collect 1.1TB high-quality Chinese data from a wide range of domains to pretrain the model. We empirically test the generation ability of PanGu-$α$ in various scenarios including text summarization, question answering, dialogue generation, etc. Moreover, we investigate the effect of model scales on the few-shot performances across a broad range of Chinese NLP tasks. The experimental results demonstrate the superior capabilities of PanGu-$α$ in performing various tasks under few-shot or zero-shot settings.

Motivation & Objective

• Motivate scaling Chinese pretrained language models beyond English-focused work.
• Develop a Transformer-based autoregressive model with an added query layer for next-token prediction.
• Construct a high-quality 1.1TB Chinese corpus from diverse sources and preprocess it for pretraining.
• Demonstrate scalable distributed training using MindSpore Auto-parallel across many devices.
• Evaluate few-shot and zero-shot performance across diverse Chinese NLP tasks.

Proposed method

• Use a uni-directional Transformer decoder with an additional query layer on top to predict the next token.
• Train PanGu-α models with 2.6B, 13B, and 200B parameters on a 1.1TB Chinese corpus.
• Apply five-dimensional parallelism (data, op-level model, pipeline model, optimizer model, rematerialization) with topology-aware scheduling in MindSpore Auto-parallel.
• Partition model and data across 2048 Ascend 910 processors with a specific sharding strategy for Q/K/V and inputs.
• Pretrain with a 40k BPE tokenizer and 1024 sequence length, using objective cross-entropy for next-token prediction.
• Assess data quality via manual and model-based evaluation, including perplexity as data-quality proxy.

Experimental results

Research questions

• RQ1 How does PanGu-α scale in parameter count and data size for Chinese language modeling?
• RQ2 Can five-dimensional Auto-parallelism enable efficient training of 200B-parameter models on large GPU/CPU clusters?
• RQ3 What is the impact of model scale on perplexity and few-shot/zero-shot performance across Chinese NLP tasks?
• RQ4 What data curation and preprocessing strategies yield high-quality Chinese pretraining data at scale?
• RQ5 What are the generation and few-shot capabilities of PanGu-α across summarization, QA, dialogue, and other tasks?

Key findings

• PanGu-α models show lower perplexity as model size increases (2.6B: 19.33; 13B: 17.69; 200B: 15.59 on validation set).
• The 200B model converges to a loss around 2.49 during training, indicating potential for further improvement with more training.
• Larger PanGu-α models achieve stronger performance in few-shot/zero-shot settings across diverse Chinese NLP tasks.
• A 1.1TB Chinese corpus is constructed from 80TB raw data, using rule-based cleaning, model-based filtering, and deduplication.
• Five-dimensional parallelism enables end-to-end training on 2048 Ascend 910 processors with topology-aware scheduling.
• The authors provide open-source Auto-parallel tools in MindSpore to facilitate similar large-scale pretraining setups.