[Paper Review] PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang+65 more|arXiv (Cornell University)|Apr 5, 2022

Topic Modeling2,124 citations

TL;DR

PaLM trains a 540B parameter autoregressive Transformer on 780B tokens using the Pathways system, achieving state-of-the-art few-shot results across hundreds of tasks, with notable gains in reasoning and multilingual capabilities.

ABSTRACT

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.

Motivation & Objective

• Investigate how scaling language models affects few-shot learning across a broad set of NLP tasks.
• Demonstrate efficient, pipeline-free training of an ultra-large model using Pathways on TPU v4 Pods.
• Evaluate PaLM on English NLP benchmarks, BIG-bench, multilingual tasks, code generation, and translation.
• Analyze bias, toxicity, and memorization, and discuss ethical considerations and potential mitigations.

Proposed method

• Use a decoder-only Transformer with SwiGLU activations and parallel transformer layers to improve training speed.
• Adopt multi-query attention, RoPE embeddings, shared input-output embeddings, no biases, and a 256k SentencePiece vocabulary to support multilingual data.
• Train three model sizes (8B, 62B, 540B) on 780B-token mixture (including code) across 6144 TPU v4 chips via the Pathways system.
• Measure model efficiency with Model FLOPs Utilization (MFU) and compare against prior models using hardware FLOPs utilization (HFU).
• Evaluate on 29 English benchmarks, BIG-bench, translation, multilingual NLG/QA, and reasoning tasks; analyze bias/toxicity and memorization.

Experimental results

Research questions

• RQ1 How does increasing model scale impact few-shot performance across diverse NLP tasks?
• RQ2 Can training a 540B model with Pathways achieve state-of-the-art results without task-specific fine-tuning?
• RQ3 What are the multilingual, reasoning, and code-generation capabilities of PaLM, and how do they scale?
• RQ4 What are the biases, toxicity, and memorization characteristics of PaLM across scales, and how can they be mitigated?

Key findings

• PaLM 540B achieves state-of-the-art few-shot results on hundreds of tasks, outperforming prior finetuned SOTA on many benchmarks.
• On BIG-bench, several tasks show discontinuous improvements with scale, indicating emergent capabilities.
• PaLM exhibits strong multilingual performance, bridging gaps with non-English SOTA on several tasks with relatively low non-English data比例.
• PaLM demonstrates breakthrough reasoning capabilities with chain-of-thought prompting in few-shot settings.
• Model FLOPs utilization (MFU) for PaLM 540B reaches 46.2% with self-attention, and MFU 45.7% without, indicating high training efficiency.
• Bias and toxicity analyses reveal scale-dependent improvements in some bias tasks but persistent stereotype associations and prompt-contingent toxicity.