[Paper Review] In-Context Learning Learns Label Relationships but Is Not Conventional Learning

Jannik Kossen, Yarin Gal+1 more|arXiv (Cornell University)|Jul 23, 2023

Innovative Teaching and Learning Methods4 citations

TL;DR

This paper investigates in-context learning (ICL) in large language models (LLMs), demonstrating that ICL leverages label relationships from in-context examples but does so differently than conventional learning. Using probabilistic metrics and controlled experiments, the authors show ICL depends on in-context labels, struggles to overcome pre-training biases, and does not treat all in-context information equally—revealing a middle ground between conventional learning and non-learning behavior.

ABSTRACT

The predictions of Large Language Models (LLMs) on downstream tasks often improve significantly when including examples of the input--label relationship in the context. However, there is currently no consensus about how this in-context learning (ICL) ability of LLMs works. For example, while Xie et al. (2021) liken ICL to a general-purpose learning algorithm, Min et al. (2022) argue ICL does not even learn label relationships from in-context examples. In this paper, we provide novel insights into how ICL leverages label information, revealing both capabilities and limitations. To ensure we obtain a comprehensive picture of ICL behavior, we study probabilistic aspects of ICL predictions and thoroughly examine the dynamics of ICL as more examples are provided. Our experiments show that ICL predictions almost always depend on in-context labels and that ICL can learn truly novel tasks in-context. However, we also find that ICL struggles to fully overcome prediction preferences acquired from pre-training data and, further, that ICL does not consider all in-context information equally.

Motivation & Objective

• To understand whether in-context learning (ICL) in large language models (LLMs) genuinely learns from in-context label relationships, as opposed to merely reflecting pre-training biases.
• To investigate whether ICL can overcome prediction preferences acquired during pre-training when in-context labels conflict with those preferences.
• To determine if ICL treats all in-context information equally, especially when label relationships are inconsistent or conflicting.
• To provide a comprehensive empirical analysis of ICL dynamics using probabilistic metrics, revealing changes in prediction confidence not captured by accuracy alone.
• To resolve conflicting claims in the literature about whether ICL constitutes a form of conventional learning or not.

Proposed method

• The authors reframe key questions about ICL behavior as statistical null hypotheses and test them empirically using controlled experiments.
• They evaluate ICL performance under randomized in-context labels to assess dependency on label information, using log-likelihood as a probabilistic metric.
• They introduce a novel task with no pre-training signal to test ICL's ability to learn truly new input-label relationships from scratch.
• They systematically vary the number of in-context examples and analyze prediction dynamics using moving averages and bootstrapped confidence intervals.
• They manipulate label relationships during inference (e.g., flipping labels at changepoints) to test whether ICL treats all in-context information equally.
• They compare multiple LLMs (e.g., LLaMA, Falcon, LLaMA-2) across diverse NLP tasks (SST-2, AG News, RTE, etc.) to ensure generalizability.

Experimental results

Research questions

• RQ1 Does ICL depend on the conditional label distribution of in-context examples, or is it independent of their labels?
• RQ2 Can ICL overcome pre-training biases when in-context labels contradict them?
• RQ3 Does ICL treat all in-context information equally, or does it favor information closer to the query?
• RQ4 How do probabilistic metrics like log-likelihood reveal ICL dynamics that accuracy alone cannot?
• RQ5 To what extent does ICL learn label relationships in a way analogous to conventional learning algorithms?

Key findings

• ICL predictions significantly depend on in-context labels: randomizing labels reduces average log-likelihood by up to -0.995 for LLaMA-2-70B across tasks, confirming label dependency.
• ICL can learn novel tasks from in-context examples, as shown by improved performance on a task with no pre-training signal, demonstrating genuine in-context learning capacity.
• ICL fails to overcome pre-training biases: when in-context labels conflict with pre-training preferences, performance plateaus and does not recover, even with additional prompting.
• ICL does not treat all in-context information equally: when label relationships are flipped or alternated, predictions differ significantly depending on the changepoint, indicating preferential use of recent or nearby examples.
• Probabilistic metrics reveal large shifts in prediction confidence that are not reflected in accuracy, highlighting the importance of using likelihood-based evaluation for ICL analysis.
• The LLaMA-2-70B model shows the most pronounced label dependency (log-likelihood drop of -0.995), while smaller models like LLaMA-7B show weaker but still significant dependence.