[Paper Review] Diverse Demonstrations Improve In-context Compositional Generalization

Itay Levy, Ben Bogin+1 more|arXiv (Cornell University)|Dec 13, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes using diverse demonstrations in in-context learning to improve compositional generalization in semantic parsing, where models must handle unseen combinations of known elements. By selecting demonstrations that collectively cover all required program structures—using a coverage-based strategy called Cover-LS—the method boosts performance by up to 23 absolute points on compositional generalization benchmarks, both in pure in-context learning and when combined with fine-tuning.

ABSTRACT

In-context learning has shown great success in i.i.d semantic parsing splits, where the training and test sets are drawn from the same distribution. In this setup, models are typically prompted with demonstrations that are similar to the input utterance. However, in the setup of compositional generalization, where models are tested on outputs with structures that are absent from the training set, selecting similar demonstrations is insufficient, as often no example will be similar enough to the input. In this work, we propose a method to select diverse demonstrations that aims to collectively cover all of the structures required in the output program, in order to encourage the model to generalize to new structures from these demonstrations. We empirically show that combining diverse demonstrations with in-context learning substantially improves performance across three compositional generalization semantic parsing datasets in the pure in-context learning setup and when combined with finetuning.

Motivation & Objective

• To address the failure of in-context learning in compositional generalization settings, where no single training example is similar to the test input.
• To improve model generalization by increasing diversity in the demonstration set, ensuring all required program structures are covered.
• To develop a method that selects demonstrations not just for similarity to the input, but for structural complementarity in covering the target program.
• To evaluate the effectiveness of diverse demonstrations in both pure in-context learning and fine-tuning setups.

Proposed method

• Proposes Cover-LS, a coverage-based retrieval method that identifies local program structures in the target output and selects demonstrations that collectively cover these structures.
• Uses model predictions to identify potential sub-structures in the target program and retrieves examples that cover them, improving structural diversity.
• Employs a beam search over candidate demonstrations to maximize coverage of predicted local structures while maintaining diversity.
• Applies a retrieval-based approach that prioritizes demonstrations with different program structures, independent of input similarity.
• Combines the method with fine-tuning by training models to use demonstrations as input, with noisy demonstrations to prevent over-reliance.
• Uses both neural and non-neural retrievers to select demonstrations, with performance evaluated on dev sets to select optimal retrieval strategies.

Experimental results

Research questions

• RQ1 Can selecting diverse demonstrations that collectively cover required program structures improve in-context learning performance on compositional generalization tasks?
• RQ2 Does coverage-based demonstration selection outperform similarity-based selection in few-shot semantic parsing?
• RQ3 How does the use of diverse demonstrations affect performance when combined with fine-tuning versus pure in-context learning?
• RQ4 Can diverse demonstrations reduce the number of examples needed for effective generalization?
• RQ5 Does structural diversity in demonstrations lead to improved performance on hard, out-of-distribution examples?

Key findings

• The Cover-LS method improved performance by up to 23.2 absolute points on the SMCalFlow-CS dataset compared to similarity-based baselines.
• On the H-CLUTRR and GeoQuery-CS datasets, diverse demonstrations led to consistent improvements across all setups, including pure in-context learning.
• The method reduced the number of demonstrations required for effective generalization, showing that diversity enables more efficient few-shot learning.
• Performance gains were especially pronounced on hard examples, indicating better generalization to unseen program structures.
• The approach achieved state-of-the-art results on multiple compositional semantic parsing benchmarks, both with and without fine-tuning.
• Coverage-based diversity (Cover-LS) outperformed diversity based solely on example dissimilarity, demonstrating the importance of structural relevance.