[Paper Review] Predictability and Surprise in Large Generative Models

Deep Ganguli, Danny Hernandez+28 more|arXiv (Cornell University)|Feb 15, 2022

Scientific Computing and Data Management22 citations

TL;DR

The paper argues that large generative models exhibit smooth general capability scaling predictably with scale, while specific capabilities and outputs emerge abruptly and inputs/outputs remain open-ended, creating deployment risks and informing policy interventions.

ABSTRACT

Large-scale pre-training has recently emerged as a technique for creating capable, general purpose, generative models such as GPT-3, Megatron-Turing NLG, Gopher, and many others. In this paper, we highlight a counterintuitive property of such models and discuss the policy implications of this property. Namely, these generative models have an unusual combination of predictable loss on a broad training distribution (as embodied in their "scaling laws"), and unpredictable specific capabilities, inputs, and outputs. We believe that the high-level predictability and appearance of useful capabilities drives rapid development of such models, while the unpredictable qualities make it difficult to anticipate the consequences of model deployment. We go through examples of how this combination can lead to socially harmful behavior with examples from the literature and real world observations, and we also perform two novel experiments to illustrate our point about harms from unpredictability. Furthermore, we analyze how these conflicting properties combine to give model developers various motivations for deploying these models, and challenges that can hinder deployment. We conclude with a list of possible interventions the AI community may take to increase the chance of these models having a beneficial impact. We intend this paper to be useful to policymakers who want to understand and regulate AI systems, technologists who care about the potential policy impact of their work, and academics who want to analyze, critique, and potentially develop large generative models.

Motivation & Objective

• Explain the four distinguishing features of large generative models (smooth general capability scaling, abrupt specific capability scaling, open-ended inputs, open-ended outputs).
• Analyze how scaling laws affect development incentives, deployment motivations, and associated safety challenges.
• Illustrate potential harms from unpredictability with novel experiments and real-world-inspired examples.
• Discuss policy interventions and governance considerations to steer model development toward beneficial outcomes.

Proposed method

• Review and synthesize scaling law literature showing power-law relations between model scale, data, compute, and loss.
• Present novel experiments on large language models to illustrate harms from unpredictability (e.g., recidivism prompts).
• Provide qualitative and quantitative examples of abrupt capability emergence and open-ended behavior.
• Analyze open-ended outputs and toxicity trends across model sizes.
• Propose policy interventions and discuss industry-academic dynamics and deployment barriers.

Experimental results

Research questions

• RQ1 Does general capability scaling follow predictable laws with scale, data, and compute?
• RQ2 Do specific capabilities emerge abruptly at certain scales, and under what conditions?
• RQ3 How do open-ended inputs and outputs affect the anticipation and mitigation of harms from large models?
• RQ4 What policy and organizational interventions could steer large-model development toward beneficial outcomes?

Key findings

• Scaling laws predict that model loss decreases with larger model size, more data, and longer training, following a power-law relationship.
• Specific capabilities can emerge abruptly at scale, showing hockey-stick gains not visible from general scaling alone.
• Open-ended inputs/domains mean unknown abilities may surface only when prompted, increasing unpredictability of harms.
• Open-ended outputs, including rising toxicity with model size, illustrate societally relevant risks that scale alongside capabilities.
• Large models exhibit biases and harms similar to or exceeding those in established risk instruments when tested on sensitive tasks like recidivism prediction.
• There are economic, scientific, and prestige motivations driving deployment, alongside barriers like cost, safety, and lack of deployment standards.