[Paper Review] How not to Lie with a Benchmark: Rearranging NLP Leaderboards

Tatiana Shavrina, Valentin Malykh|arXiv (Cornell University)|Dec 2, 2021

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper challenges the use of arithmetic mean for aggregating NLP benchmark scores, proposing geometric and harmonic means as more appropriate for averaging normalized performance metrics. It re-ranks models on GLUE, SuperGLUE, XGLUE, and XTREME, revealing that human performance remains superior and current SOTA models are 1.5–2% behind, highlighting bias in standard leaderboard rankings due to flawed averaging methods.

ABSTRACT

Comparison with a human is an essential requirement for a benchmark for it to be a reliable measurement of model capabilities. Nevertheless, the methods for model comparison could have a fundamental flaw - the arithmetic mean of separate metrics is used for all tasks of different complexity, different size of test and training sets. In this paper, we examine popular NLP benchmarks' overall scoring methods and rearrange the models by geometric and harmonic mean (appropriate for averaging rates) according to their reported results. We analyze several popular benchmarks including GLUE, SuperGLUE, XGLUE, and XTREME. The analysis shows that e.g. human level on SuperGLUE is still not reached, and there is still room for improvement for the current models.

Motivation & Objective

• To identify and address the fundamental flaw in NLP benchmark scoring: the use of arithmetic mean across tasks with varying complexity, scale, and data leakage risks.
• To demonstrate that arithmetic mean distorts model rankings by failing to penalize low performance on specific tasks.
• To propose and apply geometric and harmonic means as more appropriate aggregation methods for normalized performance scores in multi-task benchmarks.
• To re-rank top models on major NLP benchmarks (GLUE, SuperGLUE, XGLUE, XTREME) using alternative averaging techniques.
• To advocate for a redesign of benchmarking practices to ensure fairer, more reliable model evaluation in NLP.

Proposed method

• The paper computes new overall scores using geometric mean (GM) and harmonic mean (HM), defined as the Nth root of the product of scores and N divided by the sum of reciprocals, respectively.
• It applies these metrics to publicly reported model scores across all tasks in GLUE, SuperGLUE, XGLUE, and XTREME, preserving task-level performance data.
• The analysis compares rankings under arithmetic mean (AM), geometric mean, and harmonic mean to detect shifts in model ordering.
• It evaluates statistical properties—sum, variance, and standard deviation—of top models to assess consistency and robustness across tasks.
• For multilingual benchmarks like XGLUE and XTREME, it examines the impact of two-stage averaging (per language, then across tasks) on overall scores.
• It extrapolates human performance on multilingual benchmarks, noting inconsistencies in human evaluation methodology across tasks.

Experimental results

Research questions

• RQ1 How does replacing arithmetic mean with geometric or harmonic mean affect model ranking on standard NLP benchmarks?
• RQ2 To what extent do current SOTA models approach or surpass human-level performance when evaluated with more appropriate averaging methods?
• RQ3 Why does the arithmetic mean lead to biased and misleading model comparisons in multi-task NLP benchmarks?
• RQ4 How do differences in test set size, data leakage susceptibility, and task age affect the reliability of benchmark scores?
• RQ5 Can alternative aggregation methods like geometric or harmonic mean better reflect a model’s balanced performance across diverse NLP tasks?

Key findings

• On SuperGLUE, the harmonic and geometric mean rankings place human performance first, with DeBERTa and T5+Meena models ranking second and third, respectively, 1.5–2% behind humans.
• The arithmetic mean ranking on GLUE places a model in 16th position as the top performer under geometric and harmonic means, indicating significant ranking distortion.
• Human performance on SuperGLUE has a higher sum (894.40) and greater variance (130.31) than top models, suggesting inconsistent but overall superior performance across tasks.
• The top three models (DeBERTa, T5+Meena, and McAlbert+DKM) show lower variance and standard deviation than humans, indicating more consistent but less peak performance.
• In XGLUE, human-level performance is extrapolated from monolingual data, but actual human performance varies significantly across languages, undermining the benchmark’s validity.
• The study reveals that the current benchmarking paradigm creates a 'benchmark lottery' where model rankings depend heavily on the choice of averaging method, not just model capability.