[Paper Review] MetaICL: Learning to Learn In Context

Sewon Min, Michael Lewis+2 more|arXiv (Cornell University)|Oct 29, 2021

Topic ModelingComputer Science61 citations

TL;DR

MetaICL trains a language model to learn tasks from context by meta-training on a diverse set of tasks, enabling few-shot learning on unseen tasks without parameter updates or task-specific templates. It often matches or beats supervised fine-tuning and outperforms larger models.

ABSTRACT

We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set of training tasks. This meta-training enables the model to more effectively learn a new task in context at test time, by simply conditioning on a few training examples with no parameter updates or task-specific templates. We experiment on a large, diverse collection of tasks consisting of 142 NLP datasets including classification, question answering, natural language inference, paraphrase detection and more, across seven different meta-training/target splits. MetaICL outperforms a range of baselines including in-context learning without meta-training and multi-task learning followed by zero-shot transfer. We find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and that using a diverse set of the meta-training tasks is key to improvements. We also show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task, and outperforms much bigger models with nearly 8x parameters. Finally, we show that MetaICL is complementary to human-written instructions, and the best performance can be achieved by combining both approaches.

Motivation & Objective

• Motivate improving in-context learning autonomy without hand-crafted templates or task-specific reformulations.
• Leverage a large, diverse meta-training task collection to teach the model to recover task semantics from few examples.
• Evaluate performance on strictly new unseen target tasks with no overlap to meta-training tasks across multiple settings.
• Demonstrate when and how meta-training improves robustness to domain shifts and task-format diversity.

Proposed method

• Define MetaICL as meta-training a pretrained LM on many tasks using k+1 examples per task to simulate in-context learning during training.
• Train with a cross-entropy objective to predict the label of the (k+1)-th example given the prior k examples and inputs.
• Evaluate on unseen target tasks by conditioning on k labeled examples plus the test input without any parameter updates.
• Use a diverse set of 142 datasets from CrossFit and UnifiedQA to form seven meta-training/target splits with no overlap.
• Introduce Channel MetaICL, a noisy-channel variant that reparameterizes P(y|x) as P(x|y)P(y) for inference.
• Compare against strong baselines including standard in-context learning, PMI-based methods, multi-task zero-shot transfer, and fine-tuning.

Experimental results

Research questions

• RQ1 Can MetaICL enable effective in-context learning on unseen tasks without task-specific prompts or templates?
• RQ2 How do the number and diversity of meta-training tasks affect few-shot generalization to target tasks that differ in format or domain?
• RQ3 Is MetaICL robust to domain shifts and out-of-distribution target tasks compared to zero-shot and multi-task baselines?
• RQ4 Does MetaICL approach or surpass performance of full supervised fine-tuning on target tasks, and how does model size impact this gap?
• RQ5 Are human-written instructions complementary to MetaICL when combined?

Key findings

• MetaICL and Channel MetaICL consistently outperform baselines that do not use meta-training and outpace multi-task zero-shot transfer in most settings.
• Channel MetaICL achieves the best performance in the majority of seven settings, with notable gains in high-resource to low-resource and domain-shift tasks.
• MetaICL often approaches or surpasses the performance of models fully fine-tuned on the target data and can match or beat much larger models with eightfold more parameters.
• Performance benefits grow with the diversity of meta-training tasks; diverse task sets yield substantially better results than non-diverse ones.
• Incorporating human-written instructions with MetaICL yields complementary gains, but MetaICL without templates often remains competitive or superior in baseline comparisons.