[Paper Review] Instruction Tuning for Large Language Models: A Survey

Shengyu Zhang, Linfeng Dong+9 more|arXiv (Cornell University)|Aug 21, 2023

Topic Modeling96 citations

TL;DR

A comprehensive survey of instruction tuning (IT) for large language models, covering methodologies, datasets, models, multi-modal extensions, domain adaptation, efficiency, evaluation, and criticisms.

ABSTRACT

This paper surveys research works in the quickly advancing field of instruction tuning (IT), which can also be referred to as supervised fine-tuning (SFT)\footnote{In this paper, unless specified otherwise, supervised fine-tuning (SFT) and instruction tuning (IT) are used interchangeably.}, a crucial technique to enhance the capabilities and controllability of large language models (LLMs). Instruction tuning refers to the process of further training LLMs on a dataset consisting of extsc{(instruction, output)} pairs in a supervised fashion, which bridges the gap between the next-word prediction objective of LLMs and the users' objective of having LLMs adhere to human instructions. In this work, we make a systematic review of the literature, including the general methodology of SFT, the construction of SFT datasets, the training of SFT models, and applications to different modalities, domains and application, along with analysis on aspects that influence the outcome of SFT (e.g., generation of instruction outputs, size of the instruction dataset, etc). We also review the potential pitfalls of SFT along with criticism against it, along with efforts pointing out current deficiencies of existing strategies and suggest some avenues for fruitful research. Project Page: github.com/xiaoya-li/Instruction-Tuning-Survey

Motivation & Objective

• Describe the general IT pipeline and its motivation as bridging next-word prediction with instruction following.
• Survey commonly used IT datasets and their construction methods.
• Review representative instruction-tuned models and their training setups.
• Examine multi-modal IT datasets and approaches across domains and applications.
• Discuss efficiency, evaluation, criticisms, and future directions in IT.

Proposed method

• Present the standard IT pipeline including instruction, optional input, and target output.
• Explain two main data-collection approaches: annotated dataset transformation and generation via LLMs.
• Catalog major IT datasets and their characteristics (e.g., Natural Instructions, P3, xP3, Flan 2021, Unnatural Instructions, Self-Instruct, Evol-Instruct, LIMA, Super-Natural Instructions, Dolly, OpenAssistant Conversations, Baize).
• Summarize widely-used IT-tuned models and the fine-tuning setups (SFT and RLHF-like steps).
• Describe multi-modal IT extensions and datasets for images, speech, and video.
• Review efficiency enhancements and evaluation metrics, plus criticisms and open problems.

Experimental results

Research questions

• RQ1 What are the core methodologies used to construct instruction datasets for IT?
• RQ2 How do instruction-tuned models compare across prominent benchmarks and tasks?
• RQ3 What multi-modal and domain-specific adaptations exist for IT in LLMs?
• RQ4 What are the main efficiency concerns and proposed remedies in IT workflows?
• RQ5 What criticisms exist regarding IT's ability to truly learn tasks versus surface-level patterning?

Key findings

• IT datasets vary in size and source, with examples ranging from tens of thousands to millions of instances and including both human-crafted and model-generated content.
• Prominent IT models include InstructGPT, BLOOMZ, Flan-T5, Alpaca, Vicuna, GPT-4-LLM, Claude, WizardLM, Evol-Instruct, LIMA, and OpenAssistant-based variants, each with distinct fine-tuning pipelines often combining SFT and RLHF-like optimization.
• IT generally improves alignment to user instructions and controllability, with notable gains in toxicity reduction, factuality, and instruction-following in many automated and human evaluations.
• Multi-turn and multi-modal IT efforts extend instruction-following capabilities to conversations and non-text modalities, though challenges remain in efficiency and generalization to unseen tasks.
• Criticisms include limited coverage of target behaviors, reliance on surface patterns, and concerns about true task understanding beyond formatting and prompts.