[Paper Review] Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions

Swaroop Mishra, Daniel Khashabi+2 more|arXiv (Cornell University)|Apr 18, 2021

Topic Modeling33 references32 citations

TL;DR

This paper introduces NATURAL INSTRUCTIONS, a benchmark of 600k task instances with 61 natural language instructions, to evaluate how well NLP models generalize to new tasks via few-shot prompting or fine-tuning. Results show that while models like GPT-3 and BART improve with instructions, generalization remains limited, especially across diverse instruction domains, indicating significant room for improvement in instruction-following generalization.

ABSTRACT

Can we enable NLP models to appropriately respond to instructional prompts and consequently generalize to new tasks? To study this question, we leverage the existing NLP datasets and the instructions that were used to crowdsource them to create NATURAL INSTRUCTIONS, a dataset of instructions and task-specific input/output data. This dataset consists of 61 distinct language instructions and about 600k task instances, and is used to evaluate existing state-of-the-art language-models (LMs) in addressing new tasks by few-shot prompting of GPT3 and fine-tuning BART. Our analysis indicates that: (a) the existing models indeed benefit from instructions and hence, show improved generalization to new tasks; (b) while models like GPT-3 generally benefit from instructions, the extent of their gains varies across different fields of instructions and also depends on the task being solved; (c) generalization to unseen tasks in NATURAL INSTRUCTIONS remains far from perfect for the state-of-the-art, indicating significant room for more progress in this direction.

Motivation & Objective

• To investigate whether language models can generalize to new tasks when prompted with natural language instructions.
• To create a large-scale, diverse benchmark of instruction-task pairs to evaluate instruction generalization across domains.
• To assess the performance of state-of-the-art models like GPT-3 and BART on unseen tasks using few-shot prompting and fine-tuning.
• To analyze how instruction type and task domain affect model generalization performance.
• To identify gaps in current models' ability to generalize to new tasks despite instruction tuning.

Proposed method

• Leveraged existing NLP datasets and their original crowd-sourced instructions to construct NATURAL INSTRUCTIONS, a dataset of 61 distinct language instructions and ~600k input-output task instances.
• Used the dataset to evaluate few-shot prompting with GPT-3 and fine-tuning with BART on zero-shot generalization to new tasks.
• Classified instructions into distinct fields (e.g., text classification, sequence labeling) to analyze performance variation across domains.
• Evaluated model performance using standard NLP metrics on unseen task instances not seen during training.
• Compared generalization performance across different instruction types and task types to identify patterns in model behavior.

Experimental results

Research questions

• RQ1 To what extent do language models benefit from natural language instructions when generalizing to new tasks?
• RQ2 How does instruction type and task domain affect the generalization performance of NLP models?
• RQ3 How do few-shot prompting with GPT-3 and fine-tuning with BART compare in generalizing to unseen tasks in NATURAL INSTRUCTIONS?
• RQ4 What are the limitations of current models in generalizing to new tasks despite instruction tuning?
• RQ5 Are there specific instruction fields where generalization is consistently weaker or stronger?

Key findings

• Language models such as GPT-3 show measurable improvements in generalization when provided with natural language instructions, confirming the value of instruction tuning.
• The extent of performance gain from instructions varies significantly across different instruction fields and task types, indicating domain-dependent effectiveness.
• Despite gains, generalization to unseen tasks in NATURAL INSTRUCTIONS remains far from optimal, highlighting a critical gap in current model capabilities.
• Fine-tuning BART on instruction-task pairs improves performance over few-shot prompting, but generalization remains limited across diverse instruction types.
• Models exhibit inconsistent performance across instruction fields, suggesting that instruction diversity and task complexity are key challenges for generalization.
• The results indicate that while instruction following improves generalization, achieving robust zero-shot or few-shot generalization across all tasks remains an open challenge.