[Paper Review] Instruction Following without Instruction Tuning

John K. Hewitt, Nelson F. Liu+2 more|arXiv (Cornell University)|Sep 21, 2024

Education and Technology Integration4 citations

TL;DR

This paper demonstrates that language models can develop strong instruction-following capabilities not through explicit instruction tuning, but via implicit adaptation—such as response-only tuning or single-task finetuning on narrow domains like poetry. Remarkably, even a simple rule-based adapter modifying probability distributions can yield instruction following comparable to full instruction tuning, showing that instruction following emerges from minimal, non-targeted distributional adjustments.

ABSTRACT

Instruction tuning commonly means finetuning a language model on instruction-response pairs. We discover two forms of adaptation (tuning) that are deficient compared to instruction tuning, yet still yield instruction following; we call this implicit instruction tuning. We first find that instruction-response pairs are not necessary: training solely on responses, without any corresponding instructions, yields instruction following. This suggests pretrained models have an instruction-response mapping which is revealed by teaching the model the desired distribution of responses. However, we then find it's not necessary to teach the desired distribution of responses: instruction-response training on narrow-domain data like poetry still leads to broad instruction-following behavior like recipe generation. In particular, when instructions are very different from those in the narrow finetuning domain, models' responses do not adhere to the style of the finetuning domain. To begin to explain implicit instruction tuning, we hypothesize that very simple changes to a language model's distribution yield instruction following. We support this by hand-writing a rule-based language model which yields instruction following in a product-of-experts with a pretrained model. The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words' probabilities. In summary, adaptations made without being designed to yield instruction following can do so implicitly.

Motivation & Objective

• To investigate whether instruction following in language models can emerge without explicit instruction tuning.
• To examine if training solely on responses (without instructions) can yield instruction-following behavior.
• To explore whether finetuning on narrow-domain data (e.g., poetry) implicitly enables broad instruction following.
• To evaluate whether simple rule-based modifications to a model’s probability distribution can induce instruction following.
• To challenge the assumption that instruction tuning is necessary for instruction-following capabilities.

Proposed method

• Response tuning: fine-tuning a language model on response-only data from the LIMA dataset, without any corresponding instructions.
• Single-task finetuning: training on narrow-domain data such as poetry, then evaluating on diverse instruction-following benchmarks.
• Rule-based adapter: constructing a product-of-experts model where a rule-based distribution modifies the base model’s output probabilities via multiplicative normalization.
• Core equation: the final output distribution is defined as $ p_a(w\mid\mathbf{x}) = p_{\text{base}}(w\mid\mathbf{x}) p_{\text{rules}}(w\mid\mathbf{x}) / Z(\mathbf{x}) $, combining base and rule-based distributions.
• Rules include: (1) linearly increasing EOS token probability with response length, (2) uniformly adjusting probabilities of 15 key words (e.g., 'I', 'We', '<', 'Should'), and (3) penalizing repetition of previously generated tokens.
• Evaluation via AlpacaEval 2 win rate against instruction-tuned LIMA models, with ablations to isolate rule contributions.

Experimental results

Research questions

• RQ1 Can instruction following emerge in language models when trained only on responses, without any instruction tokens?
• RQ2 Does finetuning on a narrow domain (e.g., poetry) implicitly lead to instruction following on out-of-distribution tasks like recipe generation?
• RQ3 Can a simple, hand-crafted rule-based adapter induce instruction-following behavior without any learning or tuning?
• RQ4 What specific distributional changes to a language model’s output are sufficient to produce instruction following?
• RQ5 To what extent do implicit adaptations replicate the capabilities of explicit instruction tuning?

Key findings

• Response tuning alone—training on responses without instructions—achieved a 43% win rate against an instruction-tuned model on AlpacaEval 2, approaching the 50% baseline for equal performance.
• Single-task finetuning on poetry data led to instruction-following behavior on diverse tasks, including recipe generation, even when instructions were unrelated to poetry.
• A rule-based adapter with three simple rules (EOS weighting, uniform token changes, and repetition penalty) achieved a 24.4% win rate against the LIMA instruction-tuned model.
• Ablation studies showed that each of the three rules contributed significantly to performance, with individual rule ablations reducing the win rate to below 17%.
• The pretrained model already ranks correct responses higher than random responses at a rate comparable to instruction-tuned models, suggesting an implicit instruction-response mapping exists prior to tuning.
• Even minimal, non-targeted distributional adjustments—such as modifying 15 word probabilities or penalizing repetition—can yield substantial instruction-following behavior.