[Paper Review] Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava, Abhinav Rastogi+449 more|arXiv (Cornell University)|Jun 9, 2022

Topic Modeling548 citations

TL;DR

The paper introduces BIG-bench, a large, diverse benchmark of 204 tasks (from 450 authors across 132 institutions) to quantify and extrapolate language model capabilities beyond current benchmarks, comparing dense and sparse transformers across scale against human raters.

ABSTRACT

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

Motivation & Objective

• Motivate the need to understand present and near-future capabilities and limitations of language models for research and societal impact
• Develop a large-scale, difficult, diverse benchmark to reveal new capabilities and potential harms beyond existing benchmarks
• Provide a human-evaluator baseline and expert evaluations to compare with model performance across scales
• Enable extrapolation of performance as model size increases to anticipate breakthroughs and guide research directions
• Promote open contribution of tasks and transparent evaluation through an open GitHub-based workflow

Proposed method

• Create BIG-bench with 204 tasks spanning linguistics, math, common-sense reasoning, biology, physics, social bias, software development, and more
• Define an API supporting JSON and programmatic tasks for zero- and few-shot evaluation
• Evaluate dense and sparse transformer models (BIG-G, BIG-G sparse, GPT-3 class, and PaLM) across six orders of magnitude in size
• Normalize task metrics to a common 0–100 scale for aggregate analysis
• Compute calibration metrics (expected calibration error, Brier score) for model predictions across scales
• Incorporate human expert raters to establish a strong baseline and assess gaps between models and humans

Experimental results

Research questions

• RQ1 How does aggregate and task-specific performance of language models scale with model size and prompting (shots) across a diverse set of tasks?
• RQ2 Are model capabilities qualitatively different from human performance, and do scaling trends predict potential breakthroughs?
• RQ3 How do model calibration and confidence relate to scale, and how does this compare to human evaluators?
• RQ4 To what extent do model classes (dense vs sparse) differ in performance and efficiency on BIG-bench tasks?
• RQ5 What are the limits of current benchmarks in capturing future capabilities and social bias dynamics?

Key findings

• Aggregate model performance improves with size and shots but remains well below human performance on BIG-bench tasks
• Model calibration improves with scale but calibration scores remain imperfect across tasks
• Model classes (dense vs sparse) show similar performance trends, with sparsity offering some benefits
• Some tasks exhibit gradual, predictable improvements tied to knowledge/memorization; others show “breakthrough” behavior at a critical scale, often needing multi-step reasoning or brittle metrics
• Social bias tends to increase with scale in ambiguous contexts, though prompting can mitigate this effect
• Even large language models remain brittle and non-English language performance varies across tasks