[Paper Review] GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Alex Wang, Amanpreet Singh+4 more|arXiv (Cornell University)|Apr 20, 2018

Topic Modeling59 references553 citations

TL;DR

GLUE introduces a nine-task NLU benchmark and online evaluation platform with a diagnostic test suite; multi-task training with attention and ELMo transfers yields better results than single-task, yet overall performance remains far from human levels.

ABSTRACT

For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems.

Motivation & Objective

• Promote development of general, task-agnostic NLU models that can transfer knowledge across diverse tasks and domains.
• Provide a diverse, challenging suite of nine English NLU tasks built from existing datasets.
• Offer an online platform for fair, model-agnostic evaluation and comparison across tasks.
• Augment the benchmark with a diagnostic test suite to analyze linguistic capabilities and failure modes.

Proposed method

• Assemble nine single-sentence or sentence-pair NLU tasks spanning sentiment, entailment, paraphrase, and similarity.
• Adopt a model-agnostic evaluation framework that accepts any method processing single-sentence or sentence-pair inputs.
• Incorporate a diagnostic analysis dataset probing phenomena such as lexical signals, logic, and world knowledge.
• Evaluate baselines including simple sentence encoders, multi-task models, and pre-trained representations (ELMo, CoVe).
• Use macro-average scoring across tasks for overall ranking, with task-wise scores and per-task metrics.
• Provide an online leaderboard and private test data to ensure fair competition.

Experimental results

Research questions

• RQ1 Can a single model trained jointly on multiple NLU tasks outperform separate-task models on a diverse benchmark?
• RQ2 How do modern pre-training and transfer techniques (e.g., ELMo, CoVe, attention) affect performance across GLUE tasks?
• RQ3 What linguistic and reasoning capabilities do current models exhibit or fail to exhibit, as revealed by the diagnostic dataset?
• RQ4 To what extent do task-specific vs. shared representations contribute to general NLU performance?
• RQ5 What are the remaining gaps in general-purpose NLU that GLUE can help illuminate?

Key findings

• Multi-task training generally yields better aggregate performance than training separate task-specific models.
• Attention mechanisms provide gains in some settings, particularly within multi-task training, but not universally.
• ELMo embeddings improve performance over pure GloVe/CoVe baselines, especially for single-sentence tasks.
• Pre-trained sentence representations (GenSen, InferSent, DisSent) offer competitive results but often lag behind task-specific or multi-task models on GLUE.
• On several tasks (e.g., CoLA, WNLI, RTE) models still underperform relative to simple baselines or human performance, indicating room for substantial improvement.
• The diagnostic dataset reveals weaknesses in logic-driven and world-knowledge reasoning, suggesting directions for future model enhancements.