[Paper Review] Emergent Abilities of Large Language Models

Jason Lee, Yi Tay+14 more|arXiv (Cornell University)|Jun 15, 2022

Topic Modeling1,022 citations

TL;DR

This paper defines and surveys emergent abilities in large language models, showing that certain tasks only become achievable at large scales and cannot be predicted from smaller models. It catalogs emergent tasks across few-shot prompting and augmented prompting, discusses possible explanations, risks, and future directions.

ABSTRACT

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

Motivation & Objective

• Define emergent abilities in the context of pre-trained Transformer language models.
• Survey observed emergent tasks across few-shot prompting and augmented prompting methods.
• Discuss factors, risks, and future directions related to scaling and emergence in LLMs.

Proposed method

• Define emergence as abilities absent in smaller models but present in larger models.
• Plot and analyze scaling curves with training FLOPs or parameter count as the scale axis to identify phase-transition-like behavior.
• Compile and summarize emergent abilities from prior work (BIG-Bench, MMLU, TruthfulQA, etc.).
• Categorize emergent abilities by prompting paradigm (few-shot, instruction tuning, scratchpad, etc.).
• Provide a table of emergent abilities with the scale at which they emerge (training FLOPs and parameters).
• Discuss potential explanations for emergence and alternatives beyond scaling (data quality, architectures, objectives).

Experimental results

Research questions

• RQ1 What constitutes an emergent ability in large language models?
• RQ2 Which tasks become emergent as models scale, and at what scales do they appear?
• RQ3 Do augmented prompting and instruction-tuning yield emergent benefits only at large scales?
• RQ4 What mechanisms might explain emergent abilities, beyond simple scaling?

Key findings

• Emergent abilities arise only at sufficiently large model scales and cannot be predicted by extrapolating from smaller models.
• Eight examples of emergent few-shot prompting tasks span multiple model families (BIG-Bench tasks).
• Augmented prompting strategies (e.g., chain-of-thought, instruction tuning) show emergence at large scales, sometimes requiring tens to hundreds of billions of parameters or equivalent FLOPs.
• Some tasks (e.g., WiC) show emergence only at very large scales or with specific architectures (PaLM vs GPT-3/Chinchilla).
• Emergence is not solely about scale; data quality, architecture, and training objectives can influence when and how abilities emerge.
• Emergent risks accompany scaling (bias, toxicity, memorization) and require governance and mitigation strategies.