[Paper Review] Zero-Shot Text Classification with Self-Training

Ariel Gera, Alon Halfon+4 more|arXiv (Cornell University)|Oct 31, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a plug-and-play self-training method to improve zero-shot text classification using only class names and an unlabeled dataset, without requiring labeled data or domain expertise. By fine-tuning a pre-trained natural language inference model on its own high-confidence predictions, the approach achieves significant performance gains across diverse text classification tasks, demonstrating that self-training effectively adapts general-purpose zero-shot models to specific downstream tasks.

ABSTRACT

Recent advances in large pretrained language models have increased attention to zero-shot text classification. In particular, models finetuned on natural language inference datasets have been widely adopted as zero-shot classifiers due to their promising results and off-the-shelf availability. However, the fact that such models are unfamiliar with the target task can lead to instability and performance issues. We propose a plug-and-play method to bridge this gap using a simple self-training approach, requiring only the class names along with an unlabeled dataset, and without the need for domain expertise or trial and error. We show that fine-tuning the zero-shot classifier on its most confident predictions leads to significant performance gains across a wide range of text classification tasks, presumably since self-training adapts the zero-shot model to the task at hand.

Motivation & Objective

• To address the performance limitations of general-purpose zero-shot text classifiers, which often underperform compared to supervised models due to lack of task-specific adaptation.
• To explore whether self-training can effectively adapt off-the-shelf zero-shot models to downstream tasks using only unlabeled data and class names.
• To provide a practical, plug-and-play method that requires minimal effort and no labeled data, enabling practitioners to enhance model performance with minimal intervention.
• To evaluate the effectiveness of self-training in adapting entailment-based zero-shot classifiers across diverse text classification benchmarks.
• To investigate whether self-training can bridge the distributional gap between pre-trained models and downstream tasks by adapting the model to task-specific characteristics.

Proposed method

• The method uses a pre-trained natural language inference (NLI) model as a zero-shot classifier by prompting each input text with a hypothesis template that includes the class name, e.g., 'This text is about [class]' to generate entailment scores.
• The model makes predictions on an unlabeled dataset and selects the top-k examples with the highest confidence, defined as the difference between the top and second-highest prediction scores.
• These high-confidence predictions are treated as pseudo-labels and used to fine-tune the original zero-shot model in a self-training loop.
• The process is iterative: the model is retrained on its own pseudo-labeled predictions, gradually adapting to the target task’s distribution and linguistic patterns.
• Confidence-based example selection ensures that only the most reliable predictions are used for retraining, reducing noise from uncertain predictions.
• The approach is plug-and-play, requiring only class names and an unlabeled dataset—no additional hyperparameter tuning or domain-specific knowledge is needed.

Experimental results

Research questions

• RQ1 Can self-training significantly improve the performance of off-the-shelf zero-shot text classifiers without requiring labeled data for the target task?
• RQ2 Does self-training effectively adapt a general-purpose zero-shot model to the linguistic and distributional characteristics of a specific downstream task?
• RQ3 How much unlabeled data is needed to achieve meaningful performance gains through self-training in the zero-shot setting?
• RQ4 Can self-training improve performance even when the unlabeled data comes from a different but related domain or task?
• RQ5 How does self-training compare to other zero-shot adaptation techniques in terms of scalability and practical usability?

Key findings

• Self-training leads to significant performance gains across a wide range of text classification tasks, with improvements observed on standard benchmarks such as AG News, 20 Newsgroups, and DBPedia.
• The method achieves consistent gains even with as few as 10,000 unlabeled examples, demonstrating its efficiency and practicality.
• The performance boost is attributed to the model’s adaptation to the target task’s distribution, as self-training helps bridge the gap between the pre-trained model’s general knowledge and task-specific patterns.
• Cross-task self-training—using unlabeled data from related domains—also yields performance improvements, indicating that the method generalizes well across similar tasks.
• The approach is robust and effective across diverse datasets, including those with varying class distributions and text lengths, without requiring task-specific hyperparameter tuning.
• The authors release code and evaluation tools, enabling reproducibility and further research on self-training for zero-shot classification.