[Paper Review] True Few-Shot Learning with Language Models

Ethan Perez, Douwe Kiela+1 more|arXiv (Cornell University)|May 24, 2021

Topic ModelingComputer Science79 references191 citations

TL;DR

The paper shows that true few-shot model selection (prompts and hyperparameters chosen from very few labeled examples) yields only marginal gains over random prompts and often underperforms compared to held-out validation, highlighting a fundamental challenge in true few-shot learning.

ABSTRACT

Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly-selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.

Motivation & Objective

• Clarify what true few-shot learning means and why it matters for prompt and hyperparameter selection in language models.
• Empirically evaluate common model selection criteria (cross-validation and minimum description length) in true few-shot settings.
• Quantify how true few-shot prompt and hyperparameter selection compares to using held-out data.
• Investigate the stability and transferability of prompts chosen under true few-shot regimes.

Proposed method

• Formalize true few-shot learning and distinguish it from tuned and multi-distribution few-shot settings.
• Evaluate cross-validation (LOOCV) and MDL (online coding) as prompt and hyperparameter selection criteria.
• Test 9 language models (various sizes) on LAMA/UHN, plus additional tasks (RTE, CB, WiC, etc.) using multiple prompts.
• Measure performance via negative log-likelihood for prompt evaluation and accuracy for downstream tasks.
• Analyze variability, compute conservative CV (lpha) to study variance control, and examine prompt transfer across models.
• Assess computation vs. performance trade-offs in true few-shot selection.

Experimental results

Research questions

• RQ1 How well can prompts be selected in a true few-shot setting using cross-validation and MDL?
• RQ2 Do CV/MDL-prompt selections meaningfully outperform random prompts, and how do they compare to held-out validation?
• RQ3 How does true few-shot hyperparameter selection perform relative to validation-tuned hyperparameters?
• RQ4 Is prompt/hyperparameter selection reliable across model sizes and tasks?
• RQ5 Does increasing data, computation, or changing loss criteria improve true few-shot selection outcomes?

Key findings

• CV/MDL prompt selection yields only marginal improvements over random prompts and often underperforms the best prompt selected via held-out validation.
• Prompt selection becomes less reliable as model size grows, with higher risk of selecting worse-than-average prompts (long-tail effects).
• For ADAPET-based techniques, true few-shot hyperparameter selection performs at or below average and often below the best hyperparameters chosen with validation.
• Across tasks (LAMA variants and classification tasks RTE, CB, WiC), CV/MDL prompts generally underperform held-out or best prompts, with high variance in gains.
• Increasing the number of examples or computation does not consistently improve true few-shot prompt selection outcomes; variance remains high.
• The study recommends moving beyond true few-shot learning toward meta-learning, transfer/multi-task learning, data augmentation, or unsupervised model selection, and stresses transparent reporting of all hyperparameters and validation usage.