[Paper Review] Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish+8 more|arXiv (Cornell University)|Jan 23, 2020

Topic Modeling43 references1,478 citations

TL;DR

The paper empirically establishes power-law scaling laws for language model cross-entropy loss as a function of model size, dataset size, and compute, showing minimal dependence on architecture and strong, transferable scaling behavior across seven orders of magnitude.

ABSTRACT

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

Motivation & Objective

• Investigate how language model performance scales with model size, dataset size, and compute.
• Quantify the role of architecture versus scale in Transformer language models.
• Derive practical laws to predict performance and guide compute-efficient training.
• Explore overfitting behavior and the infinite data limit under scaling.
• Provide guidance on optimal allocation of a fixed compute budget for training.

Proposed method

• Train Transformer and LSTM models across a wide range of sizes (up to ~1.5B non-embedding parameters) on WebText2.
• Measure cross-entropy loss on a 1024-token context across in-distribution and transfer datasets.
• Fit observed performance to power-law scaling laws L ∝ X^−α for N (parameters), D (dataset size), and C (compute).
• Propose combined scaling law L(N,D) = [ (Nc/N)^{αN/αD} + Dc/D ]^{αD} to describe overfitting as a function of N and D.
• Define and utilize Cmin and Smin to model optimal compute usage and training steps under fixed compute budgets.
• Analyze critical batch size Bcrit and its relation to loss to optimize training efficiency.

Experimental results

Research questions

• RQ1 How does language model cross-entropy loss scale with model size, dataset size, and compute?
• RQ2 How weak is the dependence of performance on architectural shape when N (non-embedding parameters) is held fixed?
• RQ3 Can we predict optimal compute allocation for maximizing performance under a fixed compute budget?
• RQ4 What governs overfitting in the N–D plane, and how does L(N,D) behave as D → ∞?
• RQ5 How does transfer performance correlate with in-distribution performance as model size grows?

Key findings

• Performance follows a power-law with each scale factor N, D, and C when not bottlenecked by the other factors.
• Model performance depends strongly on scale and only weakly on architectural shape (depth, width, heads) within a wide range.
• Overfitting in the N–D plane is governed by the ratio N^{αN/αD}/D with αN ≈ 0.076 and αD ≈ 0.103, implying sublinear data requirements to avoid overfitting as N grows.
• Training curves exhibit predictable power-law behavior that is approximately independent of model size, enabling extrapolation to longer training times.
• Large models are more sample-efficient, requiring fewer optimization steps and data to reach similar performance; optimal compute allocation favors increasing model size under a fixed budget.
• The optimal batch size for training aligns with the critical batch size Bcrit, which scales with loss L as Bcrit(L) ≈ B*/L^{1/αB} with B* ≈ 2e8 and αB ≈ 0.21.