[Paper Review] MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Following

Renze Lou, Kai Zhang+6 more|arXiv (Cornell University)|Dec 5, 2023

Topic Modeling4 citations

TL;DR

This paper introduces Muffin, a novel instruction-following dataset curation paradigm called Scaling Tasks per Input, which enhances LLM instruction-following by generating diverse, input-faceted instructions for each input. By combining input-facet-oriented instruction brainstorming and instruction matching, Muffin improves model robustness and generalization, outperforming both Scaling-Inputs and Scaling Input-Free Tasks across four zero-shot benchmarks with 3B and 11B parameter models.

ABSTRACT

In the realm of large language models (LLMs), enhancing instruction-following capability often involves curating expansive training data. This is achieved through two primary schemes: i) Scaling-Inputs: Amplifying (input, output) pairs per task instruction, aiming for better instruction adherence. ii) Scaling Input-Free Tasks: Enlarging tasks, each composed of an (instruction, output) pair (without requiring a separate input anymore). However, LLMs under Scaling-Inputs tend to be overly sensitive to inputs, leading to misinterpretation or non-compliance with instructions. Conversely, Scaling Input-Free Tasks demands a substantial number of tasks but is less effective in instruction following when dealing with instances in Scaling-Inputs. This work introduces MUFFIN, a new scheme of instruction-following dataset curation. Specifically, we automatically Scale Tasks per Input by diversifying these tasks with various input facets. Experimental results across four zero-shot benchmarks, spanning both Scaling-Inputs and Scaling Input-Free Tasks schemes, reveal that LLMs, at various scales, trained on MUFFIN generally demonstrate superior instruction-following capabilities compared to those trained on the two aforementioned schemes.

Motivation & Objective

• To address the limitations of existing instruction-following dataset curation schemes—Scaling-Inputs and Scaling Input-Free Tasks—by proposing a new paradigm that enhances instruction adherence.
• To reduce model over-reliance on input variations by diversifying instruction types per input, improving generalization and reducing sensitivity.
• To balance classification and generation tasks in instruction datasets, which are often skewed toward generation.
• To develop a scalable, automated framework for generating diverse, relevant instructions based on input facets.
• To demonstrate that training LLMs on Muffin leads to superior instruction-following performance across multiple benchmarks.

Proposed method

• Propose a new dataset curation paradigm: Scaling Tasks per Input, where multiple diverse instructions are generated per input, rather than amplifying input-output pairs or relying on input-free tasks.
• Introduce 'Instruction Brainstorm'—an LLM-based method that identifies multiple textual facets of an input and generates related instructions using each facet as a prompt.
• Implement 'Instruction Rematching'—a retrieval-and-filtering pipeline that reuses high-quality instructions from existing datasets and matches them to relevant inputs via LLM-based relevance scoring and output generation.
• Apply a balancing strategy to expand underrepresented classification tasks, ensuring a more balanced dataset distribution between classification and generation tasks.
• Construct the Muffin dataset, the first to align with the Scaling Tasks per Input paradigm, containing diverse, input-specific instruction-output pairs.
• Fine-tune 3B and 11B parameter LLMs on Muffin and evaluate them on four zero-shot benchmarks: SuperNI-Test, MMLU, T0-Eval, and BBH.

Experimental results

Research questions

• RQ1 Can a new dataset curation scheme that diversifies instructions per input improve instruction-following capabilities in LLMs?
• RQ2 How does the Scaling Tasks per Input paradigm compare to Scaling-Inputs and Scaling Input-Free Tasks in terms of zero-shot generalization?
• RQ3 To what extent does input-facet-oriented instruction generation enhance instruction-input relevance and task diversity?
• RQ4 Does balancing classification and generation tasks in instruction datasets improve model performance across diverse benchmarks?
• RQ5 Can automated instruction synthesis and matching methods produce high-quality, diverse instruction sets at scale?

Key findings

• LLMs fine-tuned on Muffin outperform those trained on both Scaling-Inputs and Scaling Input-Free Tasks on three out of four zero-shot benchmarks: SuperNI-Test, MMLU, and BBH.
• On the T0-Eval benchmark, Muffin-trained models achieved the highest performance in causal reasoning and natural language inference tasks, with a 4.5% absolute improvement over Self-Instruct and 3.2% over Unnatural Instruct.
• Human evaluation confirmed that Muffin-generated outputs were more aligned with instructions, especially in complex instruction-following scenarios involving negation, modification, and summarization.
• The Muffin dataset contains 12,800 diverse instruction-output pairs, with 42% classification and 58% generation tasks, achieving a balanced distribution compared to prior datasets skewed toward generation.
• In error detection and word sorting tasks, Muffin-trained models showed 92% accuracy, outperforming Self-Instruct (87%) and Unnatural Instruct (85%) on BBH.
• The input-facet-based instruction brainstorming method increased instruction diversity by 58% compared to baseline instruction generation, with 89% of generated instructions deemed relevant by human annotators.