[论文解读] Self-Instruct: Aligning Language Models with Self-Generated Instructions

Yizhong Wang, Yeganeh Kordi+5 more|arXiv (Cornell University)|Dec 20, 2022

Topic Modeling被引用 117

一句话总结

Self-Instruct 自引导从语言模型本身的生成中引导指令数据，用于微调模型，在指令遵循方面实现巨大提升，且对人类标注需求极小。它在 Super-NaturalInstructions 上取得了 33% 的绝对提升，接近 InstructGPT-001 的表现。

ABSTRACT

Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning. Our code and data are available at https://github.com/yizhongw/self-instruct.

研究动机与目标

• Motivate reducing dependence on human-written instruction data for instruction-tuning.
• Develop a bootstrapping pipeline that generates, filters, and uses self-created instructions to fine-tune a pretrained LM.
• Demonstrate the effectiveness of self-generated instruction data by finetuning GPT-3 and evaluating on standard and novel tasks.
• Show that the approach approaches the performance of InstructGPT-001 and outperforms public instruction datasets.

提出的方法

• Initialize a seed set of 175 human-written tasks.
• Prompt a pretrained LM to generate new instructions and corresponding input-output instances.
• Classify generated instructions as classification vs non-classification tasks.
• Generate task instances using an input-first or output-first prompting strategy depending on task type.
• Filter low-quality or repetitive generations to build a large instruction dataset (~52K instructions, ~82K instances).
• Finetune the original LM on the generated data with multiple prompt templates to produce robust instruction-following behavior.

实验结果

研究问题

• RQ1 Can a language model generate high-quality, diverse instructions and corresponding task instances without extensive human labeling?
• RQ2 Does instruction-tuning on self-generated data improve zero-shot performance on standard instruction benchmarks compared with public datasets and InstructGPT?
• RQ3 How does the size and quality of self-generated instruction data impact instruction-following abilities on both existing benchmarks and user-oriented tasks?
• RQ4 What is the relative performance of self-instructed GPT-3 compared to instruction-tuned baselines on novel, user-oriented instructions?

主要发现

模型	# 参数	ROUGE-L
T5-LM	11B	25.7
GPT3	175B	6.8
T0	11B	33.1
GPT3 + T0 Training	175B	37.9
GPT3 Self-Instruct (Ours)	175B	39.9
InstructGPT-001	175B	40.8
Tk-Instruct	11B	46.0
GPT3 + SuperNI Training	175B	49.5
GPT3 Self-Instruct + SuperNI Training (Ours)	175B	51.6

• Finetuning GPT-3 with Self-Instruct data yields a 33.1 percentage point absolute gain on Super-NaturalInstructions over the vanilla GPT-3.
• GPT-3 Self-Instruct nearly matches InstructGPT-001 on Super-NI evaluation.
• Human evaluation on a newly authored 252 instruction set shows Self-Instruct outperforms models trained on public instruction datasets and is within 5% absolute of InstructGPT-001.
• Generating 52K instructions and 82K instances via self-generation provides diverse instruction types, with limited overlap with seed tasks.
• Data quality improvements (e.g., distillation from InstructGPT-003) further boost performance by about 10%.
• Self-Instruct data is shown to complement Super-NI data, providing additional gains when combined.