[Paper Review] SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Alex Wang, Yada Pruksachatkun+6 more|arXiv (Cornell University)|May 2, 2019

Topic Modeling67 references986 citations

TL;DR

SuperGLUE introduces eight harder NLP tasks, a public leaderboard, and a modular toolkit to push general-purpose language understanding beyond GLUE, with BERT-based baselines still lagging behind human performance.

ABSTRACT

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

Motivation & Objective

• Provide a more challenging, diverse set of English language understanding tasks than GLUE to better gauge progress in general-purpose NLP.
• Offer a public leaderboard, standardized data splits, and a modular software toolkit to facilitate fair and reproducible evaluation.
• Encourage development of sample-efficient, transfer, multitask, and self-supervised learning methods that close the gap to human performance.

Proposed method

• Proposes eight new language understanding tasks with diverse formats (including QA and coreference) and a mix of small to medium datasets.
• Provides human performance baselines and diagnostic datasets to analyze model capabilities and biases.
• Introduces a public, PyTorch-based software stack (jiant) for easy pretraining, multitask learning, and evaluation on SuperGLUE tasks.
• Recasts the evaluation protocol to ensure fair leaderboard submissions, with explicit data-use rules and attribution requirements.

Experimental results

Research questions

• RQ1 How difficult is the new SuperGLUE suite compared to GLUE for current state-of-the-art models?
• RQ2 Does scaling up pretraining, multitask learning, and transfer techniques translate to substantial gains on the SuperGLUE tasks?
• RQ3 To what extent do models exhibit linguistic, commonsense, and social biases as tasks become more challenging?
• RQ4 What is the gap between machine performance and human performance across the eight SuperGLUE tasks and diagnostics?

Key findings

• BERT-based baselines improve substantially over GLUE-era models but still lag behind human performance by about 20 points on average.
• Transferring supervision from related tasks (e.g., MultiNLI, SWAG) yields additional gains on several tasks.
• Performance remains strongest on BoolQ, CB, RTE, and WiC with smaller margins on more challenging tasks like WSC and COPA, highlighting remaining gaps in commonsense reasoning and coreference.
• Diagnostic datasets (AX b, AX g, Winogender) reveal that models, while competitive, still struggle with certain linguistic phenomena and gender bias analyses.
• SuperGLUE provides a meaningful, stricter benchmark that promotes the development of multitask, transfer, and unsupervised/self-supervised learning approaches.