[Paper Review] A Survey on In-context Learning

Qingxiu Dong, Lei Li+12 more|arXiv (Cornell University)|Dec 31, 2022

Topic Modeling245 citations

TL;DR

This paper surveys in-context learning (ICL) with formal definition, training and inference stages, demonstration design, scoring functions, analysis, evaluation, and future directions for LLMs.

ABSTRACT

With the increasing capabilities of large language models (LLMs), in-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP), where LLMs make predictions based on contexts augmented with a few examples. It has been a significant trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis. Additionally, we explore various ICL application scenarios, such as data engineering and knowledge updating. Finally, we address the challenges of ICL and suggest potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.

Motivation & Objective

• Define in-context learning and clarify its relation to related studies.
• Summarize training strategies and warmup methods that enhance ICL.
• Detail demonstration designing strategies (selection, ordering, formatting) and scoring functions.
• Analyze factors influencing ICL performance and provide theoretical insights.
• Outline evaluation resources, benchmarks, and future research directions.

Proposed method

• Formal definition and formulation of ICL based on a pretrained language model.
• Taxonomy of training (warmup) and inference (demonstration design and scoring) stages.
• Survey of demonstration design techniques including selection, ordering, and formatting.
• Comparison of scoring functions (Direct, Perplexity, Channel) for choosing outputs.
• Analysis of factors influencing ICL performance and emerging theoretical interpretations.
• Compilation of evaluation benchmarks and resources for ICL research.

Experimental results

Research questions

• RQ1 What is in-context learning and how is it formally defined for LLMs?
• RQ2 What training warmup methods can improve ICL before inference?
• RQ3 What strategies exist for designing demonstrations (selection, ordering, formatting) and how do they impact performance?
• RQ4 What scoring functions best convert model outputs into reliable predictions in ICL?
• RQ5 What factors influence ICL performance, and what do current analyses reveal about how ICL works?

Key findings

• ICL relies on learning from analogy through demonstrations without parameter updates.
• Warmup (supervised or self-supervised) can boost ICL but shows plateaus with larger training data.
• Demonstration design (selection, ordering, and formatting) significantly affects ICL performance.
• Three main scoring functions exist (Direct, Perplexity, Channel) with trade-offs in efficiency, coverage, and stability.
• Demonstration diversity, similarity to test inputs, and input-label formats strongly correlate with ICL success.
• New benchmarks (BIG-Bench, OPT-IML Bench) reveal ongoing evaluation challenges for ICL.