[Paper Review] Scaling Laws for Autoregressive Generative Modeling

Tom Henighan, Jared Kaplan+17 more|arXiv (Cornell University)|Oct 28, 2020

Multimodal Machine Learning ApplicationsComputer Science23 references149 citations

TL;DR

The paper shows that autoregressive Transformers exhibit consistent power-law plus constant scaling of cross-entropy loss across domains (language, images, video, multimodal, and math), with a nearly universal optimal model size scaling with compute and interpretable irreducible/reducible loss components.

ABSTRACT

We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\leftrightarrow$text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as $S($True$) + D_{\mathrm{KL}}($True$||$Model$)$, and the empirical scaling laws suggest a prediction for both the true data distribution's entropy and the KL divergence between the true and model distributions. With this interpretation, billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an $8 imes 8$ resolution, and we can forecast the model size needed to achieve any given reducible loss (ie $D_{\mathrm{KL}}$) in nats/image for other resolutions. We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question "Is a picture worth a thousand words?"; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.

Motivation & Objective

• Identify empirical scaling laws for cross-entropy loss across multiple data modalities using autoregressive Transformers.
• Characterize how optimal model size depends on compute budget and how loss decomposes into irreducible and reducible components.
• Quantify domain-specific scaling exponents and interpret results through information-theoretic lenses.
• Demonstrate downstream implications of scaling laws for classification, multilingual/multimodal tasks, and extrapolation in mathematics.

Proposed method

• Train decoder-only Transformer models with autoregressive cross-entropy loss across domains (language, images at multiple resolutions, video, multimodal image-text, and procedurally generated math).
• Fit loss L to a power-law plus constant form L(x)=L∞+(x0/x)^{αx} under varying x (model size N, compute C, and sometimes dataset size D).
• Estimate optimal model size Nopt(C) as a function of compute, finding Nopt ∝ C^β with β≈0.7 across domains.
• Use domain-specific encodings (pixel tokens, VQ codes) and sparse/dense attention patterns to manage context lengths and compute.
• Interpret irreducible loss L∞ as roughly the entropy of the true distribution and reducible loss as approximately D_KL(True||Model).
• Examine domain-specific phenomena such as mutual information in multimodal models and extrapolation behavior in math problems.

Experimental results

Research questions

• RQ1 Do scaling laws L = L∞ + (x0/x)^{αx} hold across language, image, video, multimodal, and math domains when using autoregressive Transformers?
• RQ2 How does optimal model size vary with compute budget across domains, and is the exponent β universal?
• RQ3 What is the interpretation of irreducible versus reducible loss in these domains, and what does it imply about data entropy and model KL divergence?
• RQ4 What downstream implications do scaling laws have for tasks like ImageNet fine-tuning, multimodal information gain, and mathematical problem solving extrapolation?
• RQ5 How do data structure and context length affect position-dependent loss and scaling behavior?

Key findings

• Scaling laws of the form L = L∞ + (x0/x)^{αx} hold across language, image, video, multimodal, and math domains.
• Optimal model size scales with compute as Nopt ∝ C^{0.7} across modalities.
• Irreducible loss L∞ tracks data entropy, while reducible loss tracks KL divergence between true and model distributions, with the latter often remaining informative even as generative loss nears L∞.
• Multimodal models show smooth scaling of mutual information and information gain with model size, and a metric Infogain increasing with scale.
• In math, extrapolation performance depends on training distribution performance rather than model size, indicating limited generalization benefits from larger models alone.
• Finetuning generative image models for ImageNet shows continued smooth scaling of classification loss with model size even after approaching irreducible loss for generation.
• Optimal aspect ratios favor deeper, narrower models for images and videos compared to language.