[论文解读] GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Alex Wang, Amanpreet Singh+4 more|arXiv (Cornell University)|Apr 20, 2018

Topic Modeling参考文献 59被引用 553

一句话总结

GLUE 引入一个 nine-task NLU 基准测试和在线评测平台，配有诊断测试套件；带注意力与 ELMo 转移的多任务训练效果优于单任务，但总体性能仍远低于人类水平。

ABSTRACT

For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems.

研究动机与目标

• Promote development of general, task-agnostic NLU models that can transfer knowledge across diverse tasks and domains.
• Provide a diverse, challenging suite of nine English NLU tasks built from existing datasets.
• Offer an online platform for fair, model-agnostic evaluation and comparison across tasks.
• Augment the benchmark with a diagnostic test suite to analyze linguistic capabilities and failure modes.

提出的方法

• Assemble nine single-sentence or sentence-pair NLU tasks spanning sentiment, entailment, paraphrase, and similarity.
• Adopt a model-agnostic evaluation framework that accepts any method processing single-sentence or sentence-pair inputs.
• Incorporate a diagnostic analysis dataset probing phenomena such as lexical signals, logic, and world knowledge.
• Evaluate baselines including simple sentence encoders, multi-task models, and pre-trained representations (ELMo, CoVe).
• Use macro-average scoring across tasks for overall ranking, with task-wise scores and per-task metrics.
• Provide an online leaderboard and private test data to ensure fair competition.

实验结果

研究问题

• RQ1 Can a single model trained jointly on multiple NLU tasks outperform separate-task models on a diverse benchmark?
• RQ2 How do modern pre-training and transfer techniques (e.g., ELMo, CoVe, attention) affect performance across GLUE tasks?
• RQ3 What linguistic and reasoning capabilities do current models exhibit or fail to exhibit, as revealed by the diagnostic dataset?
• RQ4 To what extent do task-specific vs. shared representations contribute to general NLU performance?
• RQ5 What are the remaining gaps in general-purpose NLU that GLUE can help illuminate?

主要发现

• Multi-task training generally yields better aggregate performance than training separate task-specific models.
• Attention mechanisms provide gains in some settings, particularly within multi-task training, but not universally.
• ELMo embeddings improve performance over pure GloVe/CoVe baselines, especially for single-sentence tasks.
• Pre-trained sentence representations (GenSen, InferSent, DisSent) offer competitive results but often lag behind task-specific or multi-task models on GLUE.
• On several tasks (e.g., CoLA, WNLI, RTE) models still underperform relative to simple baselines or human performance, indicating room for substantial improvement.
• The diagnostic dataset reveals weaknesses in logic-driven and world-knowledge reasoning, suggesting directions for future model enhancements.