[Paper Review] Few-Shot Learning with Siamese Networks and Label Tuning

Thomas Müller, Guillermo Pérez-Torró+1 more|arXiv (Cornell University)|Mar 28, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a Siamese network-based approach for few-shot text classification that uses label tuning (LT) to fine-tune only label embeddings, enabling efficient, shared model deployment across tasks. Despite slightly lower accuracy than full fine-tuning, LT achieves near-state-of-the-art performance with significantly reduced inference cost and high scalability, especially for large label sets, outperforming cross-attention models in speed and enabling multi-task inference with a single encoder.

ABSTRACT

We study the problem of building text classifiers with little or no training data, commonly known as zero and few-shot text classification. In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks. In this work, we show that with proper pre-training, Siamese Networks that embed texts and labels offer a competitive alternative. These models allow for a large reduction in inference cost: constant in the number of labels rather than linear. Furthermore, we introduce label tuning, a simple and computationally efficient approach that allows to adapt the models in a few-shot setup by only changing the label embeddings. While giving lower performance than model fine-tuning, this approach has the architectural advantage that a single encoder can be shared by many different tasks.

Motivation & Objective

• To address the challenge of low-resource text classification with minimal labeled data, particularly in zero-shot and few-shot learning settings.
• To explore Siamese networks as a competitive alternative to cross-attention models for few-shot text classification.
• To introduce label tuning (LT), a method that fine-tunes only label embeddings to enable model sharing across tasks.
• To evaluate the trade-off between performance and inference efficiency in few-shot learning.
• To assess the method's robustness across diverse tasks, languages, and label configurations.

Proposed method

• The model uses a symmetric Siamese network with a shared transformer encoder to embed both input texts and label texts into a shared vector space.
• Textual similarity is computed via dot product between input and label embeddings, enabling pre-computation of label embeddings for efficient inference.
• Label tuning (LT) updates only the label embeddings during fine-tuning, keeping the encoder weights frozen, thus enabling multi-task deployment with a single encoder.
• The method uses a batch softmax loss to optimize similarity scores between inputs and their correct labels while using other labels in the batch as in-batch negatives.
• Knowledge distillation is applied to recover performance loss from LT, using up to 10,000 unlabeled examples from the training set.
• The approach is evaluated across multiple languages and diverse text classification tasks, including sentiment, emotion, and review classification.

Experimental results

Research questions

• RQ1 Can Siamese networks with label tuning achieve competitive performance compared to cross-attention models in few-shot text classification?
• RQ2 Does label tuning enable efficient, scalable deployment of a single encoder across multiple classification tasks?
• RQ3 How does the performance of label tuning compare to full fine-tuning, especially with limited training examples?
• RQ4 What is the impact of label verbalization patterns (e.g., identity hypothesis vs. descriptive prompts) on model performance?
• RQ5 How do model performance and inference speed vary with label set size, text length, and presence of negation markers?

Key findings

• Siamese networks with label tuning achieve performance comparable to cross-attention models in both zero-shot and few-shot settings across diverse tasks and languages.
• Label tuning reduces inference cost to constant time relative to label count, while cross-attention models scale linearly with label set size, making Siamese networks significantly faster for large label sets.
• For 8 and 64 training examples, knowledge distillation recovers most of the performance gap between label tuning and full fine-tuning, especially on smaller label sets.
• Siamese networks outperform cross-attention models on shorter texts (≤44 tokens), while cross-attention models perform better on longer sequences (>160 tokens).
• Both models struggle with neutral sentiment prediction on Twitter-based datasets, but cross-attention models show higher error rates on neutral labels.
• The choice of label verbalization (e.g., identity hypothesis) has no significant impact on performance, suggesting plain label names are sufficient.