[Paper Review] Holistic Evaluation of Language Models

Percy Liang, Rishi Bommasani+48 more|arXiv (Cornell University)|Nov 16, 2022

Topic Modeling115 citations

TL;DR

HELM introduces a holistic benchmark for language models, using a taxonomy of scenarios and metrics, evaluating 30 models across 16 core scenarios with 7 metrics plus 7 targeted evaluations, and releasing prompts, completions, and a modular toolkit.

ABSTRACT

Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models. First, we taxonomize the vast space of potential scenarios (i.e. use cases) and metrics (i.e. desiderata) that are of interest for LMs. Then we select a broad subset based on coverage and feasibility, noting what's missing or underrepresented (e.g. question answering for neglected English dialects, metrics for trustworthiness). Second, we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios when possible (87.5% of the time). This ensures metrics beyond accuracy don't fall to the wayside, and that trade-offs are clearly exposed. We also perform 7 targeted evaluations, based on 26 targeted scenarios, to analyze specific aspects (e.g. reasoning, disinformation). Third, we conduct a large-scale evaluation of 30 prominent language models (spanning open, limited-access, and closed models) on all 42 scenarios, 21 of which were not previously used in mainstream LM evaluation. Prior to HELM, models on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: now all 30 models have been densely benchmarked on the same core scenarios and metrics under standardized conditions. Our evaluation surfaces 25 top-level findings. For full transparency, we release all raw model prompts and completions publicly for further analysis, as well as a general modular toolkit. We intend for HELM to be a living benchmark for the community, continuously updated with new scenarios, metrics, and models.

Motivation & Objective

• Define a taxonomy of LM evaluation scenarios and metrics to specify the design space.
• Select a broad, feasible core set of scenarios and metrics with explicit caveats about missing coverage.
• Provide dense multi-metric evaluation across scenarios to reveal trade-offs beyond accuracy.
• Standardize evaluation conditions to enable direct, fair comparisons across models.
• Deliver a living benchmark with open data, prompts, and tooling for community-driven expansion.

Proposed method

• Develop a top-down taxonomy over scenarios (task-domain-language) and seven metric categories (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency).
• Implement 16 core scenarios with 7 metrics each, plus 7 targeted evaluations over 26 additional scenarios.
• Benchmark 30 prominent language models from open, limited-access, and closed families under standardized prompting (5-shot).
• Provide public access to raw prompts and completions and a modular toolkit to add new scenarios, models, metrics, and prompting strategies.
• Analyze results to surface interactions between scenarios, metrics, and models and identify 25 high-level findings.

Figure 1 : Language model. A language model takes text (a prompt) and generates text (a completion) probabilistically. Despite their simple interface, language models can be adapted to a wide range of language tasks from question answering to summarization.

Experimental results

Research questions

• RQ1 What is the performance of current language models across a broad, standardized set of core scenarios and metrics?
• RQ2 How do accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency interact across models and scenarios?
• RQ3 What are the trade-offs and dependencies between model access (open vs closed) and evaluation outcomes?
• RQ4 How sensitive are model evaluations to prompting choices and adaptation methods?
• RQ5 What insights emerge from targeted evaluations on knowledge, reasoning, memorization, disinformation, and other specialized skills?

Key findings

• Instruction-tuning benefits appear across core metrics, with instruction-tuned models achieving strong accuracy, robustness, and fairness relative to size.
• A notable gap remains between open and non-open models in core scenarios, though open models are narrowing this gap over time.
• Calibration relationships with accuracy vary by scenario and prompting strategy.
• Robustness and fairness perturbations correlate with accuracy, yet trade-offs exist where the most accurate models are not the most robust or fair.
• Demographic metadata reveals performance disparities across dialects and groups for several models.
• Bias and toxicity generation are generally low on core scenarios but can be non-trivial in certain contexts, necessitating targeted evaluations.
• Accuracy and efficiency do not show a uniform trade-off across all models; larger models improve accuracy but incur higher costs, with some models on the accuracy-efficiency Pareto frontier.
• Question answering exhibits substantial cross-scenario heterogeneity, with text-davinci-002 often most accurate among core QA scenarios.
• Information retrieval using LMs for passage ranking can beat classical methods but may lag state-of-the-art fine-tuned retrievers under some settings.
• Memory and copyright leakage risks correlate with model accuracy, being more pronounced in higher-accuracy models.
• Knowledge-intensive tasks favor larger, or specially tuned models for high performance; code-focused models excel in reasoning tasks.
• Prompting format and adaptation strongly influence results, highlighting the challenge of standardizing evaluations across models.
• Scaling effects exist within model families, but cross-family comparisons show that some smaller, well-tuned models rival larger ones in accuracy.
• Disinformation-generation capabilities are strongest in largest models under certain prompts, with mixed results across other prompts.
• The benchmark reveals comprehensive insights beyond prior evaluations, including prompting sensitivity and downstream behavior not captured in traditional benchmarks.

Figure 2 : The importance of the taxonomy to HELM. Previous language model benchmarks (e.g. SuperGLUE, EleutherAI LM Evaluation Harness, BIG-Bench) are collections of datasets, each with a standard task framing and canonical metric, usually accuracy ( left ). In comparison, in HELM we take a top-dow