[Paper Review] Is In-Context Learning Sufficient for Instruction Following in LLMs?

Hao Zhao, Maksym Andriushchenko+2 more|arXiv (Cornell University)|May 30, 2024

Online Learning and Analytics4 citations

TL;DR

This paper investigates whether in-context learning (ICL) alone can effectively align large language models (LLMs) for instruction following, evaluating the Urial prompting method across multiple base models. Despite improvements via greedy example selection, ICL still underperforms instruction fine-tuning on benchmarks like MT-Bench and AlpacaEval 2.0, especially with more capable models, revealing fundamental limitations in ICL's ability to fully capture instruction-following behavior without parameter updates.

ABSTRACT

In-context learning (ICL) allows LLMs to learn from examples without changing their weights: this is a particularly promising capability for long-context LLMs that can potentially learn from many examples. Recently, Lin et al. (2024) proposed URIAL, a method using only three in-context examples to align base LLMs, achieving non-trivial instruction following performance. In this work, we show that, while effective, ICL alignment with URIAL still underperforms compared to instruction fine-tuning on the established benchmark MT-Bench, especially with more capable base LLMs. We then uncover the most relevant elements for successful in-context alignment, finding the crucial role of the decoding parameters. Based on these insights, we show that the approach of URIAL can indeed be improved by adding high-quality, potentially carefully selected via greedy search, demonstrations in context, getting closer to the performance of instruct models. Finally, we provide the first, to our knowledge, systematic comparison of ICL and instruction fine-tuning (IFT) for instruction following in the low data regime, where ICL can be a viable alternative to IFT. Overall, our work advances the understanding of ICL as an alignment technique and its relationship to IFT. We provide our code at https://github.com/tml-epfl/icl-alignment.

Motivation & Objective

• To assess whether in-context learning (ICL) via the Urial prompting method can achieve instruction-following performance comparable to supervised fine-tuning (SFT) in base LLMs.
• To investigate why ICL, despite its flexibility and weight-free adaptation, fails to close the performance gap with fine-tuned models on instruction-following benchmarks.
• To analyze the impact of in-context demonstration quantity, quality, and structure—especially correct answer alignment—on ICL effectiveness for instruction following.
• To evaluate the transferability of ICL demonstrations across different LLM architectures, particularly newer or larger models not used in initial Urial example selection.
• To understand whether ICL for instruction following behaves fundamentally differently than for other NLP tasks like classification or translation.

Proposed method

• Evaluating the Urial prompting strategy on multiple base LLMs, including Llama-2, Llama-3, Mistral, and GPT-4-Base, using established benchmarks such as MT-Bench and AlpacaEval 2.0 (LC).
• Testing many-shot in-context learning by adding high-quality instruction-following demonstrations from existing datasets to extend the Urial prompt beyond three examples.
• Proposing a greedy selection algorithm to identify in-context examples that maximize the MT-Bench score, improving performance without changing model weights.
• Conducting ablation studies on demonstration structure, including cases with only questions, incorrect answers, or randomly sampled answers, to isolate the role of correct answer alignment.
• Assessing transferability of selected ICL examples across different base models (e.g., Llama-2-7B-80k, Mixtral-8x22B) to evaluate model-specific sensitivity of effective demonstrations.
• Analyzing the impact of multi-turn vs. single-turn demonstrations, and why 2nd-turn performance remains significantly worse under ICL compared to fine-tuned models.

Figure 1: The results of scaling experiments with different sampling strategies on Mistral-7B-v0.2-32k. The black dashed line refers to the 1st-turn MT-Bench score of Mistral-7B-Instruct-v0.2. ICL with more demonstrations does not bridge the gap between the base model and its aligned counterpart. Su

Experimental results

Research questions

• RQ1 Does increasing the number of in-context demonstrations improve instruction-following performance in long-context LLMs, as seen in tasks like translation or summarization?
• RQ2 Can a greedy selection strategy for in-context examples close the performance gap between in-context learning and supervised fine-tuning on instruction-following benchmarks?
• RQ3 Why does in-context learning for instruction following differ fundamentally from ICL in tasks like classification or multiple choice, particularly regarding the necessity of correct answer pairs?
• RQ4 To what extent do effective in-context demonstrations generalize across different base LLMs, especially newer or larger models not used in the original Urial example selection?
• RQ5 What role do incorrect or poorly structured demonstrations play in degrading ICL performance for instruction following?

Key findings

• Adding more in-context demonstrations beyond three does not systematically improve instruction-following performance on MT-Bench, unlike in tasks such as summarization or translation.
• The greedy selection of in-context examples improves performance significantly, nearly closing the gap between Mistral-7B-v0.2 and its instruction-fine-tuned counterpart on MT-Bench (from 6.27 to 7.68 score).
• Correct answer alignment is crucial: omitting answers or using incorrect or randomly sampled answers drastically reduces performance, with circular shifts of answers dropping scores to 1.80.
• Using in-domain random answers improves performance (6.24), but out-of-domain random answers degrade it (4.13), indicating that semantic relevance of answers matters.
• The most effective ICL demonstrations do not generalize well across models: applying the same selected examples to Llama-2-7B-80k and Mixtral-8x22B-v0.1-4bit yields inconsistent improvements or even performance drops.
• Incorrect demonstrations severely degrade performance, with circular shifts of answers reducing the MT-Bench score to 1.80, highlighting the sensitivity of ICL to demonstration quality in instruction-following tasks.

Figure 2: The distribution of the 1st-turn MT-Bench score (GPT-4-Turbo as judge) on Mistral-7B-v0.2-32k obtained by adding 100 instructions from Evol-Instruct-70k as a 4th demonstration to Urial . The black dashed line refers to the 1st-turn MT-Bench score of the base model with Urial . A majority o