[Paper Review] Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns+5 more|arXiv (Cornell University)|Sep 7, 2020

Topic Modeling23 references269 citations

TL;DR

The paper introduces a 57-task multitask benchmark to measure zero-shot and few-shot language model understanding across diverse domains, showing large models like GPT-3 improve beyond random chance but still fall short of expert-level performance and exhibit calibration and knowledge gaps.

ABSTRACT

We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

Motivation & Objective

• Assess breadth and depth of pretraining-derived knowledge across diverse real-world domains.
• Evaluate zero-shot and few-shot performance to approximate human learning.
• Identify model blind spots and calibration shortcomings across tasks.

Proposed method

• Create a 57-task multiple-choice benchmark spanning humanities, social sciences, STEM, and other subjects.
• Evaluate zero-shot and few-shot performance for large language models (GPT-3 variants) and transfer-fine-tuned models (UnifiedQA).
• Use four GPT-3 sizes (Small to X-Large) and UnifiedQA with a text-to-text backbone.
• Provide per-task dev, validation, and test splits to measure transfer capabilities and avoid spurious cues.

Experimental results

Research questions

• RQ1 How well do pretrained language models acquire and apply broad world knowledge across diverse subjects without task-specific fine-tuning?
• RQ2 How does model size and prompting strategy (zero-shot vs. few-shot) affect performance across the 57 tasks?
• RQ3 What are the common failure modes and calibration properties of current models on this broad benchmark?
• RQ4 To what extent can specialized fine-tuning (UnifiedQA) close the gap relative to few-shot GPT-3 across tasks?

Key findings

• GPT-3 175B in few-shot mode achieves 43.9% average accuracy, well above random but far from expert-level performance on every task.
• UnifiedQA (11B) achieves 48.9% average accuracy, outperforming GPT-3 variants despite fewer parameters.
• Model performance is highly lopsided across tasks, with strong results on some subjects and near-random on others like mathematics, physics, law, and morality.
• GPT-3’s confidence estimates are poorly calibrated, with up to 24 percentage points discrepancy between confidence and accuracy in zero-shot settings.
• Even the best models fail to reach expert-level accuracy on any individual task, highlighting broad knowledge gaps and limitations in procedural reasoning.
• Calculation-heavy and normative-law-m morality tasks are particularly challenging for current models.