[Paper Review] Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou+33 more|arXiv (Cornell University)|Oct 20, 2022

Topic ModelingComputer Science1,182 citations

TL;DR

The paper shows that instruction finetuning scales with more tasks and larger models, and that including chain-of-thought data dramatically boosts reasoning, achieving state-of-the-art results (e.g., Flan-PaLM 540B) and strong open-ended generation quality.

ABSTRACT

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models.

Motivation & Objective

• Motivate generalization to unseen tasks by instruction-based finetuning.
• Investigate how the number of finetuning tasks impacts performance across model sizes.
• Assess the impact of incorporating chain-of-thought data into finetuning on reasoning tasks.
• Demonstrate cross-model applicability of instruction finetuning across PaLM, T5, and U-PaLM families.
• Evaluate usability and responsible AI aspects of instruction-finetuned models.

Proposed method

• Finetune multiple model families (T5, PaLM, U-PaLM) on a large mix of instruction-tuned tasks totaling 1,836 tasks from Muffin, T0-SF, NIV2, and CoT data.
• Use packing to combine multiple training examples into a single sequence with an end-of-sequence token.
• Prepend instruction templates to inputs and apply masking to boundaries; employ Adafactor optimizer and a constant learning-rate schedule.
• Experiment with zero-shot, few-shot, and chain-of-thought (CoT) prompting setups during evaluation.
• Incorporate a dedicated CoT finetuning mix of nine datasets with manually authored CoT annotations to study reasoning impact.
• Evaluate on held-out benchmarks (MMLU, BBH, TyDiQA, MGSM) and open-ended generation with human evaluation.
• Compare various model sizes (8B, 62B, 540B) and model families (Flan-T5, Flan-PaLM, cont-PaLM, U-PaLM).

Experimental results

Research questions

• RQ1 Does instruction finetuning scale benefits with increasing task count and model size?
• RQ2 How does including chain-of-thought data in finetuning affect reasoning capabilities on held-out tasks?
• RQ3 Can CoT finetuning be combined with non-CoT tasks without degrading performance on non-CoT tasks?
• RQ4 Do instruction-finetuned models generalize across architectures and pre-training objectives?
• RQ5 What is the practical impact of instruction finetuning on open-ended generation usability and responsible AI metrics?

Key findings

• Instruction finetuning yields large performance gains across model sizes and prompts, with gains ranging from 9.4% to 15.5% on held-out benchmarks.
• Increasing finetuning tasks improves performance, though gains saturate after about 282 tasks for the shown scales.
• Scaling model size from 8B to 540B provides substantial performance improvements for both finetuned and non-finetuned models.
• Incorporating nine CoT datasets in finetuning enables robust CoT reasoning across benchmarks, achieving state-of-the-art results (e.g., Flan-PaLM 540B with CoT + Self-Consistency on MMLU at 75.2%).
• Joint finetuning on CoT and non-CoT data maintains non-CoT performance while significantly boosting CoT performance.
• CoT prompting with self-consistency yields strong gains and enables zero-shot CoT reasoning on challenging tasks.
• Flan models outperform their non-instruction-finetuned counterparts across many tasks, including strong zero-shot and few-shot abilities.