[Paper Review] Embroid: Unsupervised Prediction Smoothing Can Improve Few-Shot Classification

Neel Guha, Mayee F. Chen+4 more|arXiv (Cornell University)|Jul 20, 2023

Topic Modeling4 citations

TL;DR

This paper proposes Embroid, a weakly supervised method that improves few-shot text classification by refining language model predictions using unlabeled data. It leverages prediction consistency across k-nearest neighbors in multiple embedding spaces, combining them via a latent variable model to correct mispredictions—achieving average F1 gains of 7.3 points on GPT-JT and enabling a 1.3B model to outperform a 6.7B instruction-tuned model.

ABSTRACT

Recent work has shown that language models' (LMs) prompt-based learning capabilities make them well suited for automating data labeling in domains where manual annotation is expensive. The challenge is that while writing an initial prompt is cheap, improving a prompt is costly -- practitioners often require significant labeled data in order to evaluate the impact of prompt modifications. Our work asks whether it is possible to improve prompt-based learning without additional labeled data. We approach this problem by attempting to modify the predictions of a prompt, rather than the prompt itself. Our intuition is that accurate predictions should also be consistent: samples which are similar under some feature representation should receive the same prompt prediction. We propose Embroid, a method which computes multiple representations of a dataset under different embedding functions, and uses the consistency between the LM predictions for neighboring samples to identify mispredictions. Embroid then uses these neighborhoods to create additional predictions for each sample, and combines these predictions with a simple latent variable graphical model in order to generate a final corrected prediction. In addition to providing a theoretical analysis of Embroid, we conduct a rigorous empirical evaluation across six different LMs and up to 95 different tasks. We find that (1) Embroid substantially improves performance over original prompts (e.g., by an average of 7.3 points on GPT-JT), (2) also realizes improvements for more sophisticated prompting strategies (e.g., chain-of-thought), and (3) can be specialized to domains like law through the embedding functions.

Motivation & Objective

• To address the high cost of manual prompt tuning in low-resource NLP tasks by improving predictions without labeled data.
• To develop a prompt-patching method that corrects erroneous predictions using only unlabeled data and weak supervision.
• To ensure the method is theoretically explainable, fast, and robust to avoid worsening performance.
• To evaluate the method across diverse models, tasks, and prompting strategies, including chain-of-thought and specialized domains like law.

Proposed method

• Embroid computes k-nearest neighbors for each sample using N different embedding functions to identify semantically similar samples.
• For each embedding space, it computes a scaled-modified majority vote over the LM’s predictions for neighboring samples to generate smoothed neighborhood votes.
• It combines the original LM prediction with the N smoothed votes using a latent variable graphical model estimated via a fast method-of-moments estimator.
• The method uses a thresholding strategy where votes are set to ±1 based on whether neighborhood vote averages exceed the global mean for that source.
• It is designed to be robust by aggregating signals across multiple embedding spaces, reducing reliance on any single imperfect representation.
• The framework is compatible with various prompting strategies, including chain-of-thought and few-shot prompting, and can be adapted to domain-specific embeddings.

Figure 1 : The Embroid method for prompt-patching.

Experimental results

Research questions

• RQ1 Can prediction smoothing via neighborhood consistency improve few-shot classification without labeled data?
• RQ2 How does the performance of Embroid depend on the quality of the embedding functions and the base prompt accuracy?
• RQ3 Does Embroid improve performance across diverse language models and prompting strategies, including chain-of-thought and advanced prompting techniques?
• RQ4 Can Embroid be effectively applied to specialized domains like law using domain-specific embeddings?
• RQ5 How does Embroid compare to existing weakly supervised methods like AMA in terms of performance and robustness?

Key findings

• Embroid improves F1 by an average of 7.3 points on GPT-JT across 95 tasks, with gains observed even on smaller models.
• The method enables a 1.3B parameter model to outperform a 6.7B instruction-tuned model, demonstrating significant efficiency gains.
• Embroid improves performance on GPT-3.5 by an average of 4.9 F1 points per task, with minimal risk of degradation.
• The method achieves consistent improvements across six different language models, including GPT-3.5, J1-Jumbo, OPT, Bloom, and GPT-JT.
• Embroid is complementary to advanced prompting strategies, improving performance when applied to chain-of-thought, AMA, and selective annotation prompts.
• The method is robust to hyperparameter choices, with the default setting of τ⁺ᵢ = τ⁻ᵢ = E[λᵢ] achieving near-optimal performance without tuning.