[Paper Review] With Little Power Comes Great Responsibility

Dallas Card, Peter Henderson+4 more|arXiv (Cornell University)|Oct 13, 2020

Topic Modeling63 references4 citations

TL;DR

This paper highlights the widespread issue of underpowered experiments in NLP, demonstrating that small test sets in benchmarks like GLUE and typical human evaluation designs often lack sufficient statistical power to reliably detect meaningful model improvements. The authors conduct power analyses across multiple NLP tasks—showing, for example, that 2,000-sentence MT test sets have only ~75% power to detect 1 BLEU-point differences—and provide open-source notebooks to promote rigorous, power-aware experimental design in future NLP research.

ABSTRACT

Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community. Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements, and increase the chances of exaggerated findings. By meta-analyzing a set of existing NLP papers and datasets, we characterize typical power for a variety of settings and conclude that underpowered experiments are common in the NLP literature. In particular, for several tasks in the popular GLUE benchmark, small test sets mean that most attempted comparisons to state of the art models will not be adequately powered. Similarly, based on reasonable assumptions, we find that the most typical experimental design for human rating studies will be underpowered to detect small model differences, of the sort that are frequently studied. For machine translation, we find that typical test sets of 2000 sentences have approximately 75% power to detect differences of 1 BLEU point. To improve the situation going forward, we give an overview of best practices for power analysis in NLP and release a series of notebooks to assist with future power analyses.

Motivation & Objective

• To investigate the prevalence of low statistical power in NLP experiments, which undermines the reliability of reported model improvements.
• To demonstrate that common NLP evaluation practices—especially small test sets and under-sampled human evaluations—often fail to detect true model differences.
• To quantify the actual statistical power of existing experimental designs in NLP, including model accuracy comparisons and machine translation evaluations.
• To advocate for and provide practical tools (e.g., open-source notebooks) to integrate power analysis into standard NLP experimental workflows.
• To address the replication crisis in NLP by reducing Type-M and Type-S errors from underpowered studies.

Proposed method

• Conduct meta-analyses of existing NLP papers and datasets to estimate typical effect sizes, sample sizes, and variances across tasks.
• Use mixed-effects modeling to account for variability across workers and items in human evaluation studies, enabling accurate power estimation.
• Simulate statistical power under various assumptions (e.g., effect size, significance level, sample size) using Monte Carlo methods.
• Apply power analysis to machine translation using BLEU scores, modeling differences as a function of test set size.
• Develop and release open-source Jupyter notebooks to assist researchers in conducting power analyses for their own experiments.
• Use contrast coding in mixed models to estimate differences between models relative to a grand mean, improving interpretability of effect sizes.

Experimental results

Research questions

• RQ1 What is the typical statistical power of experiments in the GLUE benchmark, particularly for comparisons against state-of-the-art models?
• RQ2 How underpowered are typical human evaluation studies in NLP when detecting small model improvements?
• RQ3 What sample size is required to achieve adequate power (≥80%) for detecting a 1 BLEU-point difference in machine translation?
• RQ4 To what extent do underpowered experiments lead to exaggerated or misleading effect size estimates in NLP research?
• RQ5 How can researchers systematically incorporate power analysis into their experimental design to improve reproducibility and validity?

Key findings

• For several tasks in the GLUE benchmark, such as MRPC and SST-2, existing test sets are too small to provide adequate statistical power for detecting meaningful model improvements.
• Human evaluation studies using typical sample sizes (e.g., 25–100 raters or items) are often underpowered to detect small but meaningful differences between models.
• In machine translation, a standard test set of 2,000 sentences has approximately 75% power to detect a 1 BLEU-point difference, falling short of the conventional 80% threshold.
• The authors find that significant results from underpowered experiments are prone to Type-M (magnitude) and Type-S (sign) errors, leading to overestimation or reversal of true effect sizes.
• The study reveals a general lack of statistical evaluation and poor reporting in recent NLP papers, exacerbating the risk of unreliable findings.
• The authors release open-source notebooks to enable researchers to perform power analyses for their own experiments, promoting more rigorous and reproducible evaluation practices.