[Paper Review] The Natural Language Decathlon: Multitask Learning as Question Answering

Bryan McCann, Nitish Shirish Keskar+2 more|arXiv (Cornell University)|Jun 20, 2018

Topic Modeling106 references339 citations

TL;DR

This paper introduces decaNLP, a ten-task multitask benchmark reframed as question answering, and MQAN, a single multitask QA model with a multi-pointer-generator decoder that excels across tasks and enables transfer and zero-shot capabilities.

ABSTRACT

Deep learning has improved performance on many natural language processing (NLP) tasks individually. However, general NLP models cannot emerge within a paradigm that focuses on the particularities of a single metric, dataset, and task. We introduce the Natural Language Decathlon (decaNLP), a challenge that spans ten tasks: question answering, machine translation, summarization, natural language inference, sentiment analysis, semantic role labeling, zero-shot relation extraction, goal-oriented dialogue, semantic parsing, and commonsense pronoun resolution. We cast all tasks as question answering over a context. Furthermore, we present a new Multitask Question Answering Network (MQAN) jointly learns all tasks in decaNLP without any task-specific modules or parameters in the multitask setting. MQAN shows improvements in transfer learning for machine translation and named entity recognition, domain adaptation for sentiment analysis and natural language inference, and zero-shot capabilities for text classification. We demonstrate that the MQAN's multi-pointer-generator decoder is key to this success and performance further improves with an anti-curriculum training strategy. Though designed for decaNLP, MQAN also achieves state of the art results on the WikiSQL semantic parsing task in the single-task setting. We also release code for procuring and processing data, training and evaluating models, and reproducing all experiments for decaNLP.

Motivation & Objective

• Propose a unified multitask benchmark (decaNLP) by casting ten diverse NLP tasks as question answering over a context.
• Develop MQAN, a single multitask QA network, without task-specific modules, that jointly learns all decaNLP tasks.
• Demonstrate transfer learning, domain adaptation, and zero-shot capabilities arising from multitask training on decaNLP.
• Show that anti-curriculum training further improves multitask performance and that MQAN achieves strong single-task results in semantic parsing.

Proposed method

• Recast ten NLP tasks (QA, translation, summarization, NLI, sentiment, SRL, relation extraction, dialog, semantic parsing, pronoun resolution) as QA problems with context, question, and answer triplets.
• Propose MQAN with dual coattention encoder, and a multi-pointer-generator decoder that jointly attends to context and question and can generate from an external vocabulary, copy from context, or copy from question.
• Train MQAN on all decaNLP tasks together with an anti-curriculum strategy and analyze its zero-shot and transfer capabilities.
• Evaluate using task-specific metrics mapped to a decaScore (0-100 per task) and aggregate across ten tasks.
• Provide open-source code and data processing pipelines for reproducibility and leaderboard tracking.

Experimental results

Research questions

• RQ1 Can a single model trained on ten diverse NLP tasks framed as QA achieve competitive performance across all tasks without task-specific modules?
• RQ2 Does multitask learning on decaNLP yield transfer learning benefits to new domains, languages, or related tasks beyond the training distribution?
• RQ3 How does the multi-pointer-generator decoder contribute to performance across tasks requiring copying, generation, and question-based disambiguation?
• RQ4 What is the impact of curriculum vs anti-curriculum training strategies on the decaNLP performance and stability?

Key findings

• MQAN trained on decaNLP achieves competitive performance across multiple tasks in both multitask and single-task settings.
• The multi-pointer-generator decoder enables effective copying from context and question and generation from an external vocabulary, which is crucial for tasks with varying output needs.
• Anti-curriculum training improves decaNLP performance beyond fully joint training alone.
• MQAN pretrained on decaNLP shows transfer gains for machine translation and named entity recognition, domain adaptation for sentiment analysis and natural language inference, and zero-shot capabilities for text classification.
• MQAN attains state-of-the-art results on WikiSQL semantic parsing in the single-task setting.