[Paper Review] Structured Prompting: Scaling In-Context Learning to 1,000 Examples

Yaru Hao, Yutao Sun+4 more|arXiv (Cornell University)|Dec 13, 2022

Topic Modeling11 citations

TL;DR

This paper introduces structured prompting, a method that scales in-context learning to 1,000 examples by grouping demonstrations, encoding them independently with right-aligned position embeddings, and using rescaled attention to efficiently attend to all exemplars. The approach reduces computational complexity from quadratic to linear, significantly improving performance and stability over conventional in-context learning.

ABSTRACT

Large language models have exhibited intriguing in-context learning capability, achieving promising zero- and few-shot performance without updating the parameters. However, conventional in-context learning is usually restricted by length constraints, rendering it ineffective to absorb supervision from a large number of examples. In order to go beyond few shots, we introduce structured prompting that breaks the length limit and scales in-context learning to thousands of examples. Specifically, demonstration examples are separately encoded with well-designed position embeddings, and then they are jointly attended by the test example using a rescaled attention mechanism. So we can scale the number of exemplars with linear complexity instead of quadratic complexity with respect to length. Experimental results on a diverse set of tasks show that our approach improves end-task performance and reduces evaluation variance over conventional in-context learning as the number of demonstration examples increases. Code has been released at https://aka.ms/structured-prompting.

Motivation & Objective

• To overcome the length and computational limitations of conventional in-context learning, which typically restricts demonstrations to 5–100 examples due to context window and quadratic attention complexity.
• To improve the stability and performance of in-context learning by scaling the number of demonstration examples beyond few-shot regimes.
• To enable efficient, linear-complexity attention over large numbers of exemplars without parameter updates or fine-tuning.
• To reduce performance variance caused by example order and permutation in few-shot in-context learning.

Proposed method

• Divide N demonstration examples into M groups, each independently encoded using a decoder-only transformer with right-aligned position embeddings to maintain consistent relative distances to the test input.
• Cache only the key and value vectors of self-attention for each group, enabling efficient retrieval during inference.
• Use a rescaled attention mechanism that normalizes attention scores between the test input and grouped exemplars, balancing context and input contributions.
• Concatenate the encoded group representations with the test input and pass through the transformer decoder for autoregressive generation.
• Apply different alignment strategies—such as truncation, attention masking, or padding space—to align grouped prompts with the test input, with truncation showing best results on BLOOM.
• Use a fixed template to convert input-output pairs into semantically meaningful demonstrations before grouping.

Experimental results

Research questions

• RQ1 Can in-context learning be effectively scaled to thousands of examples without parameter updates or fine-tuning?
• RQ2 How can attention complexity be reduced from quadratic to linear when handling large numbers of demonstrations?
• RQ3 Does increasing the number of exemplars improve both performance and stability in in-context learning?
• RQ4 Which alignment strategy (e.g., truncation, attention masking) best preserves model performance when grouping multiple exemplars?
• RQ5 How does structured prompting compare to conventional in-context learning in terms of variance and accuracy across diverse NLP tasks?

Key findings

• Structured prompting achieves state-of-the-art performance on multiple NLP tasks, including text classification, multi-choice, and open-ended generation, outperforming conventional in-context learning.
• On the BLOOM-7B model, structured prompting with the 'truncate' strategy achieves 72.5% average accuracy across tasks, compared to 70.9% without right-alignment.
• The method reduces performance variance caused by example permutation, with consistent gains as the number of exemplars increases.
• On the FairseqLM-6.7B model, structured prompting achieves 75.6% average accuracy, showing robustness across different model sizes.
• The 'truncate' strategy consistently outperforms other alignment methods, especially on BLOOM, suggesting it best preserves attention balance.
• The approach enables effective use of up to 1,000 examples in context, significantly exceeding the typical 5–100 example limit of standard in-context learning.