[Paper Review] Instruction Pre-Training: Language Models are Supervised Multitask Learners

Daixuan Cheng, Yuxian Gu+4 more|arXiv (Cornell University)|Jun 20, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces Instruction Pre-Training, a method that enhances language model pre-training by augmenting raw corpora with 200 million synthetic instruction-response pairs generated via an open-source-based instruction synthesizer. The approach enables models like Llama3-8B to match or exceed Llama3-70B performance in continual pre-training, while also improving generalization and instruction tuning gains.

ABSTRACT

Unsupervised multitask pre-training has been the critical method behind the recent success of language models (LMs). However, supervised multitask learning still holds significant promise, as scaling it in the post-training stage trends towards better generalization. In this paper, we explore supervised multitask pre-training by proposing Instruction Pre-Training, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train LMs. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of Instruction Pre-Training. In pre-training from scratch, Instruction Pre-Training not only consistently enhances pre-trained base models but also benefits more from further instruction tuning. In continual pre-training, Instruction Pre-Training enables Llama3-8B to be comparable to or even outperform Llama3-70B. Our model, code, and data are available at https://github.com/microsoft/LMOps.

Motivation & Objective

• To explore the potential of supervised multitask pre-training as a scalable alternative to unsupervised pre-training in language models.
• To address the challenge of scaling supervised multitask learning by generating high-quality, diverse instruction-response pairs from raw text.
• To develop a cost-effective instruction synthesizer based on open-source models to enable large-scale data augmentation.
• To evaluate the effectiveness of Instruction Pre-Training in both pre-training from scratch and continual pre-training on domain-specific data.
• To demonstrate that instruction-augmented pre-training enhances downstream generalization and instruction tuning performance.

Proposed method

• Augment raw pre-training corpora with synthetic instruction-response pairs generated by a fine-tuned instruction synthesizer.
• Train the instruction synthesizer on curated, diverse datasets where each example includes raw text and multiple instruction-response pairs conditioned on that text.
• Use a 7B-parameter open-source language model as the base for the instruction synthesizer to ensure cost efficiency and scalability.
• Synthesize 200 million instruction-response pairs across 40+ task categories by applying the trained synthesizer to massive raw text corpora.
• Pre-train language models on the augmented corpora using standard causal language modeling objectives.
• Apply continual pre-training on domain-specific data (e.g., finance, biomedicine) using the instruction-augmented corpora to improve domain adaptation.

Experimental results

Research questions

• RQ1 Can supervised multitask pre-training, when scaled effectively, outperform or match unsupervised pre-training in language model performance?
• RQ2 How effective is a synthesizer based on open-source models in generating high-quality, diverse instruction-response pairs for pre-training data augmentation?
• RQ3 To what extent does Instruction Pre-Training improve generalization and downstream instruction tuning performance?
• RQ4 Can instruction-augmented pre-training enable smaller models like Llama3-8B to achieve performance comparable to larger models like Llama3-70B in domain-specific settings?
• RQ5 What are the key factors in data quality and diversity that contribute to the success of instruction-augmented pre-training?

Key findings

• A 500M-parameter model pre-trained via Instruction Pre-Training on 100B tokens achieved performance comparable to a 1B-parameter model pre-trained on 300B tokens.
• Models pre-trained with Instruction Pre-Training showed significantly greater gains from further instruction tuning compared to standard pre-training.
• In continual pre-training, Llama3-8B fine-tuned on instruction-augmented data matched or outperformed Llama3-70B on finance and biomedicine benchmarks.
• The instruction synthesizer, trained on diverse datasets, generalized effectively to unseen raw text, enabling scalable and high-quality data synthesis.
• The synthetic instruction-response pairs covered 40+ task categories with high knowledge coverage and correctness, ensuring effective multitask learning.
• The method demonstrated strong transferability and scalability, with results validated across multiple domains and model sizes.