[Paper Review] Efficient Few-Shot Learning Without Prompts

Lewis Tunstall, Nils Reimers+5 more|arXiv (Cornell University)|Sep 22, 2022

Topic Modeling98 citations

TL;DR

SetFit is a prompt-free, Siamese-style fine-tuning framework for sentence transformers that enables strong few-shot text classification with much smaller models and faster training/inference than prompting-based methods.

ABSTRACT

Recent few-shot methods, such as parameter-efficient fine-tuning (PEFT) and pattern exploiting training (PET), have achieved impressive results in label-scarce settings. However, they are difficult to employ since they are subject to high variability from manually crafted prompts, and typically require billion-parameter language models to achieve high accuracy. To address these shortcomings, we propose SetFit (Sentence Transformer Fine-tuning), an efficient and prompt-free framework for few-shot fine-tuning of Sentence Transformers (ST). SetFit works by first fine-tuning a pretrained ST on a small number of text pairs, in a contrastive Siamese manner. The resulting model is then used to generate rich text embeddings, which are used to train a classification head. This simple framework requires no prompts or verbalizers, and achieves high accuracy with orders of magnitude less parameters than existing techniques. Our experiments show that SetFit obtains comparable results with PEFT and PET techniques, while being an order of magnitude faster to train. We also show that SetFit can be applied in multilingual settings by simply switching the ST body. Our code is available at https://github.com/huggingface/setfit and our datasets at https://huggingface.co/setfit .

Motivation & Objective

• Address limitations of prompt-based and large-model few-shot methods (prompt engineering, high compute, instability).
• Propose SetFit as a prompt-free, two-step fine-tuning framework for sentence transformers in few-shot settings.
• Demonstrate SetFit’s performance across diverse datasets, languages, and distillation scenarios.
• Compare SetFit against state-of-the-art PET/PEFT-based methods and prompt-based approaches in terms of accuracy and efficiency.
• Provide publicly available code and data to facilitate adoption in practical few-shot tasks.

Proposed method

• Fine-tune a pretrained Sentence Transformer (ST) on small labeled sets using a contrastive Siamese objective to produce rich embeddings.
• Generate augmented training data for contrastive fine-tuning via positive and negative sentence triplets across classes (R positive, R negative per class).
• Encode labeled examples with the fine-tuned ST to create embeddings, then train a logistic regression classification head on these embeddings.
• Perform inference by encoding new inputs with the ST and predicting with the trained head.
• Use cosine-similarity loss for ST fine-tuning with learning rate 1e-3, batch size 16, max length 256 tokens, for 1 epoch.

Experimental results

Research questions

• RQ1 Can a prompt-free, sentence-transformer-based approach achieve competitive accuracy in few-shot learning compared to PET/PEFT and prompt-based methods?
• RQ2 How does SetFit perform across multilingual data and various text classification tasks with limited labeled data?
• RQ3 What are the efficiency (training/inference) advantages of SetFit relative to large-scale prompt-based approaches?
• RQ4 Is SetFit effective in few-shot distillation and under resource-constrained hardware conditions?

Key findings

• SetFit MPNet achieves strong few-shot performance, outperforming FineTune by an average of 19.3 points at N=8 and narrowing the gap at N=64.
• SetFit MPNet outperforms Perfect and Adapet at small N and is competitive with T-Few 3B while being prompt-free and significantly smaller.
• On RAFT, SetFit RoBERTa surpasses GPT-3 and PET and even exceeds the human baseline on several tasks, though it trails T-Few 11B.
• In multilingual MARC experiments, SetFit with multilingual ST embeddings consistently outperforms FineTune and Adapet in the 8-shot regime across languages.
• Distillation experiments show a SetFit teacher-student setup where a smaller SetFit student can match or exceed a baseline student with limited unlabeled data, especially at very small N.
• SetFit offers an order-of-magnitude faster training/inference and much smaller storage footprint than T-Few 3B, with substantial practical deployment advantages.