[Paper Review] Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps

Fuxiao Liu, Paiheng Xu+3 more|arXiv (Cornell University)|Jul 11, 2023

Topic Modeling5 citations

TL;DR

This paper investigates in-context learning (ICL) in large language models (LLMs) using contrastive demonstrations and saliency maps to understand which parts of input-label pairs drive predictions. It finds that label flipping reduces saliency of ground-truth labels, input neutralization has lesser impact than label flipping, and complementary explanations do not consistently improve performance—highlighting the role of pretrained knowledge and task-specific design in ICL effectiveness.

ABSTRACT

We investigate the role of various demonstration components in the in-context learning (ICL) performance of large language models (LLMs). Specifically, we explore the impacts of ground-truth labels, input distribution, and complementary explanations, particularly when these are altered or perturbed. We build on previous work, which offers mixed findings on how these elements influence ICL. To probe these questions, we employ explainable NLP (XNLP) methods and utilize saliency maps of contrastive demonstrations for both qualitative and quantitative analysis. Our findings reveal that flipping ground-truth labels significantly affects the saliency, though it's more noticeable in larger LLMs. Our analysis of the input distribution at a granular level reveals that changing sentiment-indicative terms in a sentiment analysis task to neutral ones does not have as substantial an impact as altering ground-truth labels. Finally, we find that the effectiveness of complementary explanations in boosting ICL performance is task-dependent, with limited benefits seen in sentiment analysis tasks compared to symbolic reasoning tasks. These insights are critical for understanding the functionality of LLMs and guiding the development of effective demonstrations, which is increasingly relevant in light of the growing use of LLMs in applications such as ChatGPT. Our research code is publicly available at https://github.com/paihengxu/XICL.

Motivation & Objective

• To understand the mechanisms behind in-context learning (ICL) in large language models (LLMs) by analyzing the contribution of different demonstration components.
• To investigate how contrastive demonstrations—via label flipping, input neutralization, and added explanations—affect model predictions and saliency patterns.
• To evaluate whether complementary explanations enhance ICL performance across different tasks, especially in sentiment classification.
• To compare saliency patterns across models of varying scale (e.g., GPT-2 vs. InstructGPT) in response to demonstration perturbations.
• To provide actionable insights for practitioners on curating effective demonstrations for LLMs, especially in light of the growing use of models like ChatGPT.

Proposed method

• Construct contrastive demonstrations by flipping ground-truth labels, neutralizing sentiment-indicative terms in inputs, and adding complementary explanations to original examples.
• Use gradient-based (Integrated Gradients) and perturbation-based (LIME) saliency map methods to quantify token-level importance in model predictions.
• Perform qualitative and quantitative analysis of saliency maps across original and contrastive demonstrations to compare feature importance.
• Reproduce prior findings on label flipping and model scale (e.g., Min et al., 2022; Wei et al., 2023) in a sentiment classification task using the SST-2 dataset.
• Apply T-tests to assess statistical significance of differences in averaged saliency scores between original and perturbed components across 20 test examples.
• Use LIME with 5 neighbors per example to generate local, instance-level explanations, acknowledging limitations due to computational and API constraints.

Experimental results

Research questions

• RQ1 How does flipping the ground-truth label in demonstrations affect the saliency of label tokens in LLM predictions?
• RQ2 To what extent does neutralizing sentiment-indicative terms in input text reduce their contribution to model predictions?
• RQ3 Do complementary explanations in demonstrations consistently improve ICL performance on sentiment classification tasks?
• RQ4 How do saliency patterns differ between smaller models (e.g., GPT-2) and larger models (e.g., InstructGPT) under the same perturbations?
• RQ5 What role does pretrained knowledge play in driving predictions when demonstration components are perturbed?

Key findings

• After label flipping, ground-truth labels in demonstrations show significantly reduced saliency, with p < 0.001 in a T-test across 20 test examples, indicating that models no longer rely on the flipped labels.
• Neutralizing sentiment-indicative terms in inputs results in lower averaged saliency scores compared to original terms, with p < 0.001 for GPT-2, suggesting models still rely on pretrained knowledge.
• For InstructGPT, the difference in saliency between original and neutralized terms is less pronounced (p = 0.17), likely due to sparse saliency maps with many zeros from LIME.
• Complementary explanations have similar or higher averaged saliency scores than original review tokens in 16 out of 20 examples for GPT-2, but do not improve performance on sentiment classification.
• The saliency of explanation tokens is 90% of the saliency of review tokens on average, indicating explanations are as important as inputs in attention, yet they do not enhance performance.
• Larger models like InstructGPT show emergent ability to override prior knowledge when labels are flipped, while smaller models like GPT-2 do not, consistent with prior findings.