[Paper Review] Instruction Tuning with GPT-4

Baolin Peng, Chunyuan Li+3 more|arXiv (Cornell University)|Apr 6, 2023

Topic Modeling186 citations

TL;DR

The paper uses GPT-4 to generate 52K English and Chinese instruction-following data to fine-tune LLaMA models, achieving strong zero-shot instruction-following performance and comparable results to GPT-4 on unseen tasks, with release of data and code.

ABSTRACT

Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed. In this paper, we present the first attempt to use GPT-4 to generate instruction-following data for LLM finetuning. Our early experiments on instruction-tuned LLaMA models show that the 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks to the instruction-following data generated by previous state-of-the-art models. We also collect feedback and comparison data from GPT-4 to enable a comprehensive evaluation and reward model training. We make our data generated using GPT-4 as well as our codebase publicly available.

Motivation & Objective

• Motivate instruction tuning of open-source LLMs using machine-generated data.
• Investigate whether GPT-4-generated instruction data improves zero-shot generalization over prior data.
• Explore cross-language (English/Chinese) instruction-following capabilities.
• Develop reward models from GPT-4 comparisons to guide evaluation and decoding.
• Provide publicly available GPT-4-generated data and tuning pipelines to the research community.

Proposed method

• Generate English instruction-following data for 52K Alpaca instructions using GPT-4 (one response per instruction).
• Translate instructions to Chinese via ChatGPT and answer in Chinese to create Chinese instruction-following data.
• Create comparison data by having GPT-4 rate and compare model outputs to train a reward model.
• Train LLaMA-7B models on GPT-4 generated data to obtain LLaMA-GPT4 and LLaMA-GPT4-CN.
• Use three evaluation setups: human alignment (HHH criteria), GPT-4-based automatic evaluation, and ROUGE-L on Unnatural Instructions.
• Release GPT-4 generated data and the finetuned model checkpoints and codebase.

Experimental results

Research questions

• RQ1 Does GPT-4 generated instruction data improve zero-shot generalization on unseen tasks compared to prior data sources?
• RQ2 Can instruction-tuning with GPT-4 data achieve performance close to GPT-4 on unseen instructions?
• RQ3 How does cross-language (English vs Chinese) instruction-following generalize for open-source LLMs?
• RQ4 Can a GPT-4–driven reward model effectively guide evaluation and decoding of instruction-following outputs?
• RQ5 What are the practical considerations for data scale, model size, and RLHF integration in this pipeline?

Key findings

• GPT-4-generated instruction data yields superior zero-shot performance on unseen tasks compared with GPT-3.5-based data.
• LLaMA-GPT4 (7B) often outperforms 13B Alpaca and LLaMA baselines in automatic evaluations against strong models like ChatGPT and GPT-4.
• GPT-4–instruction-tuned LLaMA closely matches GPT-4 in alignment criteria across human evaluation, suggesting GPT-4 data can rival the teacher model for certain tasks.
• Cross-language results show that Chinese instruction-following data (from GPT-4) enables effective Chinese models and enables cross-language generalization analyses.
• GPT-4 based reward modeling provides consistent ranking signals for decoding and evaluation, aligning with human and GPT-4 feedback.