[論文レビュー] A Survey on In-context Learning

Qingxiu Dong, Lei Li+12 more|arXiv (Cornell University)|Dec 31, 2022

Topic Modeling被引用数 245

ひとこと要約

この論文は、形式的定義、トレーニングと推論の段階、デモンストレーション設計、スコアリング関数、分析、評価、および大規模言語モデル（LLM）の今後の方向性における文脈内学習（ICL）を概観します。

ABSTRACT

With the increasing capabilities of large language models (LLMs), in-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP), where LLMs make predictions based on contexts augmented with a few examples. It has been a significant trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis. Additionally, we explore various ICL application scenarios, such as data engineering and knowledge updating. Finally, we address the challenges of ICL and suggest potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.

研究の動機と目的

• Define in-context learning and clarify its relation to related studies.
• Summarize training strategies and warmup methods that enhance ICL.
• Detail demonstration designing strategies (selection, ordering, formatting) and scoring functions.
• Analyze factors influencing ICL performance and provide theoretical insights.
• Outline evaluation resources, benchmarks, and future research directions.

提案手法

• Formal definition and formulation of ICL based on a pretrained language model.
• Taxonomy of training (warmup) and inference (demonstration design and scoring) stages.
• Survey of demonstration design techniques including selection, ordering, and formatting.
• Comparison of scoring functions (Direct, Perplexity, Channel) for choosing outputs.
• Analysis of factors influencing ICL performance and emerging theoretical interpretations.
• Compilation of evaluation benchmarks and resources for ICL research.

実験結果

リサーチクエスチョン

• RQ1 What is in-context learning and how is it formally defined for LLMs?
• RQ2 What training warmup methods can improve ICL before inference?
• RQ3 What strategies exist for designing demonstrations (selection, ordering, formatting) and how do they impact performance?
• RQ4 What scoring functions best convert model outputs into reliable predictions in ICL?
• RQ5 What factors influence ICL performance, and what do current analyses reveal about how ICL works?

主な発見

• ICL relies on learning from analogy through demonstrations without parameter updates.
• Warmup (supervised or self-supervised) can boost ICL but shows plateaus with larger training data.
• Demonstration design (selection, ordering, and formatting) significantly affects ICL performance.
• Three main scoring functions exist (Direct, Perplexity, Channel) with trade-offs in efficiency, coverage, and stability.
• Demonstration diversity, similarity to test inputs, and input-label formats strongly correlate with ICL success.
• New benchmarks (BIG-Bench, OPT-IML Bench) reveal ongoing evaluation challenges for ICL.