[Paper Review] Learning and Evaluating General Linguistic Intelligence

Dani Yogatama, Cyprien de Masson d’Autume+9 more|arXiv (Cornell University)|Jan 31, 2019

Topic Modeling29 references156 citations

TL;DR

The paper defines general linguistic intelligence for language tasks, proposes an online prequential coding metric to measure rapid adaptation, and empirically audits state-of-the-art models for transfer, generalization, and forgetting across multiple NLP tasks. It finds that current models still require substantial in-domain data, suffer catastrophic forgetting, and overfit to specific datasets rather than solving general linguistic tasks.

ABSTRACT

We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of the test data that quantifies how quickly an existing agent (model) learns a new task. Our results show that while the field has made impressive progress in terms of model architectures that generalize to many tasks, these models still require a lot of in-domain training examples (e.g., for fine tuning, training task-specific modules), and are prone to catastrophic forgetting. Moreover, we find that far from solving general tasks (e.g., document question answering), our models are overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss missing components and conjecture on how to make progress toward general linguistic intelligence.

Motivation & Objective

• Define general linguistic intelligence as the ability to reuse prior linguistic knowledge to rapidly adapt to new tasks.
• Evaluate state-of-the-art NLP models for task independence and knowledge reuse.
• Propose and validate an online prequential coding metric to quantify rapid learning on new tasks.
• Analyze transfer learning, generalization across datasets, and catastrophic forgetting in continual learning settings.

Proposed method

• Use two main model classes: Transformer-based (BERT) and recurrent (ELMo+BiDAF) architectures.
• Pretrain models on unsupervised objectives and/or other supervised tasks, then fine-tune on target tasks like SQuAD and MNLI.
• Introduce online codelength based on prequential coding to measure how many training examples are needed for a model to learn a new task.
• Evaluate generalization by testing SQuAD-trained models on TriviaQA, QuAC, QA-SRL, and QA-ZRE datasets.
• Examine continual learning by training on sequences of tasks (unsupervised→SQuAD→MNLI/TriviaQA) and study forgetting and curriculum effects.
• Discuss potential improvements through memory modules, meta-learning, and improved curricula.

Experimental results

Research questions

• RQ1 How quickly do existing models adapt to new linguistic tasks given prior training?
• RQ2 To what extent do pretrained models generalize across datasets within the same task without additional fine-tuning?
• RQ3 How does pretraining on related tasks influence sample efficiency and final performance on target tasks?
• RQ4 What is the impact of training curricula on continual learning and forgetting of previously learned tasks?
• RQ5 Can memory- or meta-learning approaches improve robustness to domain shifts and catastrophic forgetting?

Key findings

• Pretraining significantly reduces the amount of in-domain data needed to reach high performance but 40k examples are still needed to approach asymptotic performance on SQuAD and MNLI.
• Models pretrained on other supervised tasks often yield better online codelengths, indicating faster practical learning despite similar final accuracy.
• High-performing SQuAD models do not generalize well to TriviaQA, QuAC, QA-SRL, or QA-ZRE without additional training data from those datasets.
• Continual learning leads to rapid forgetting of earlier tasks, even when task-specific final layers exist, highlighting the need for better transfer and memory mechanisms.
• A random multitask curriculum can mitigate forgetting and achieve competitive performance across tasks, but may require seeing all tasks from the start or retraining.
• The paper discusses potential improvements including elastic weight consolidation, memory modules, and meta-learning as avenues toward general linguistic intelligence.