[Paper Review] LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

Minghao Wu, Abdul Waheed+3 more|arXiv (Cornell University)|Apr 27, 2023

Topic Modeling25 citations

TL;DR

LaMini-LM proposes a diverse set of small instruction-tuned models (61M–7B) distilled from large LLMs using a 2.58M-instruction dataset, achieving competitive performance with much smaller sizes and providing analysis on hallucination and toxicity.

ABSTRACT

Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities. However, these models are resource-intensive. To alleviate this issue, we explore distilling knowledge from instruction-tuned LLMs into much smaller ones. To this end, we carefully develop a large set of 2.58M instructions based on both existing and newly-generated instructions. In addition to being sizable, we design our instructions to cover a broad set of topics to ensure diversity. Extensive analysis of our instruction dataset confirms its diversity, and we generate responses for these instructions using gpt-3.5-turbo. Leveraging these instructions, we fine-tune a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both the encoder-decoder and decoder-only families, with varying sizes. We evaluate the performance of our models using automatic metrics on 15 different natural language processing (NLP) benchmarks, as well as through human assessment. The results demonstrate that our proposed LaMini-LM models are comparable to competitive baselines, while being much smaller in size.

Motivation & Objective

• Motivate the development of smaller, efficient instruction-tuned models to mitigate resource demands of LLMs.
• Create a large, diverse instruction dataset (LaMini instruction dataset) totaling 2.58M examples.
• Distill knowledge from LLMs into a diverse set of smaller models across architectures and sizes.
• Evaluate performance on multiple NLP benchmarks with both automatic and human evaluation; assess hallucination and toxicity.
• Provide dataset, model checkpoints, and code to facilitate further research and deployment in resource-constrained settings.

Proposed method

• Assemble a 2.58M instruction-response dataset by combining self-instruct, P3, FLAN, Alpaca sources and augment with example-guided and topic-guided generation using gpt-3.5-turbo.
• Perform sequence-level (offline) distillation by training small student models to imitate the outputs of the teacher (gpt-3.5-turbo) on the generated dataset.
• Fine-tune a family of models (61M–7B) across encoder-decoder and decoder-only architectures with initialization from multiple sources (T5, Flan-T5, Cerebras-GPT, GPT-2/Neo/J, LLaMA).
• Evaluate on 15 diverse NLP benchmarks using an automatic evaluation harness and conduct human evaluation on 114 user-oriented instructions.
• Assess hallucination via a dedicated LaMini-Hallucination test set and toxicity via RealToxicityPrompts with OpenAI Moderation.
• Analyze the impact of dataset subsets and architecture (encoder-decoder vs decoder-only) on performance.

Experimental results

Research questions

• RQ1 Can a large-scale distilled instruction dataset yield competitive performance in small models?
• RQ2 How do different model architectures and sizes affect instruction-tuned performance after distillation?
• RQ3 What is the effect of using different subsets of the instruction dataset on downstream tasks and human evaluation?
• RQ4 How do LaMini-LMs perform in terms of hallucination and toxicity compared to baselines and proprietary models?

Key findings

• Many LaMini models match or surpass baselines like Alpaca-7B and even approach LLaMA-7B on downstream tasks while remaining significantly smaller.
• Encoder-decoder LaMini models often outperform decoder-only counterparts at smaller scales; LaMini-Flan-T5-248M can match LLaMA-7B, and LaMini-LLaMA-7B outperforms LLaMA-7B and Alpaca-7B in aggregate results.
• The 2.58M LaMini instruction dataset substantially influences performance; subsets derived from P3/FLAN improve downstream tasks, while self-instruct-related data boosts human evaluation scores.
• Despite improvements, there remains a gap between open-source LaMini models and GPT-3.5-turbo in hallucination handling, prompting the creation of the LaMini-Hallucination benchmark.
• Toxicity analysis shows encoder-decoder models generally generate less toxic outputs before instruction tuning, but post-tuning trends vary by architecture, indicating nuanced behavior across model families.