[Paper Review] In-context Example Selection with Influences

Tai D. Nguyen, Eric Wong|arXiv (Cornell University)|Feb 21, 2023

Topic Modeling4 citations

TL;DR

This paper introduces in-context influences—a method to quantify the impact of individual in-context examples on few-shot language model performance. By leveraging retraining-based influence estimation via forward passes, the approach identifies high-impact (positive) and low-impact (negative) examples, outperforming baselines on 9 SuperGLUE tasks with up to a 16.3% performance gain when selecting top-influence examples over bottom-influence ones.

ABSTRACT

In-context learning (ICL) is a powerful paradigm emerged from large language models (LLMs). Despite its promises, ICL performance is known to be highly sensitive to input examples. In this work, we use $ extit{in-context influences}$ to analyze few-shot ICL performance directly from the in-context examples. Our proposed influence-based example selection method can identify both positive and negative examples, outperforming several baselines when evaluated on 9 SuperGLUE tasks. Our analysis uncovers up to a $16.3\%$ performance gap between using the most negative in-context examples compared to the most positive. In a case study, we apply our influence-based framework to quantify the phenomena of recency bias in example ordering for few-shot ICL.

Motivation & Objective

• To understand and quantify the impact of individual in-context examples on few-shot in-context learning (ICL) performance.
• To develop a scalable, offline method for selecting optimal in-context examples based on their influence on model predictions.
• To analyze the variability in ICL performance due to example selection and ordering, especially under known biases like recency and majority bias.
• To generalize the influence framework beyond classification to other NLP tasks and model families.

Proposed method

• The method uses a retraining-based influence framework adapted to ICL, where 'training' is simulated via forward passes on prompts with subsets of in-context examples.
• It constructs a dataset of k-shot prompts with varying subsets of examples and measures their validation performance to estimate influence.
• Influence scores are computed by learning a linear mapping from subset performance to individual example contributions, using a dev set of examples.
• The framework generalizes to any performance metric and can be applied to study positional effects, such as recency bias, by analyzing influence across example orderings.
• It leverages query-access to the LLM, avoiding gradient computation, making it computationally efficient compared to standard influence methods.

Figure 1: Test accuracy increases when examples are selected in increasing in-context influence percentile bins.

Experimental results

Research questions

• RQ1 How do individual in-context examples influence few-shot ICL performance, and can we quantify their positive or negative impact?
• RQ2 Can influence-based example selection outperform existing baselines like semantic similarity or perplexity in selecting effective in-context examples?
• RQ3 What is the magnitude of performance difference between the most influential and least influential examples in ICL?
• RQ4 How does example ordering affect ICL performance, and can influence analysis quantify recency bias?
• RQ5 To what extent do in-context influences correlate with known signals like example perplexity or embedding similarity?

Key findings

• Influence-based example selection outperforms multiple baselines in both positive and negative example selection across 9 SuperGLUE tasks.
• The performance gap between the most positive and most negative in-context examples is up to 16.3% on LLaMA-13B, demonstrating significant variability in example impact.
• The framework successfully quantifies recency bias, showing that later-positioned examples have higher influence, especially in longer k-shot settings.
• Influence scores show little correlation with known signals such as example perplexity or embedding similarity, indicating they capture a distinct, informative signal.
• The method generalizes across multiple model families and tasks, including multi-choice and binary classification, with stronger gains observed on multi-choice tasks.

Figure 2: Token budget comparison for different baselines evaluated on LLaMA-7B ( $|S|=400$ ).