[Paper Review] Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach

Wenpeng Yin, Jamaal Hay+1 more|arXiv (Cornell University)|Aug 31, 2019

Topic ModelingComputer Science25 references34 citations

TL;DR

This paper benchmarks zero-shot text classification (0shot-tc) across multiple aspects (topic, emotion, situation), standardizes datasets and evaluations, and proposes a textual entailment framework to handle unseen labels without task-specific training.

ABSTRACT

Zero-shot text classification (0Shot-TC) is a challenging NLU problem to which little attention has been paid by the research community. 0Shot-TC aims to associate an appropriate label with a piece of text, irrespective of the text domain and the aspect (e.g., topic, emotion, event, etc.) described by the label. And there are only a few articles studying 0Shot-TC, all focusing only on topical categorization which, we argue, is just the tip of the iceberg in 0Shot-TC. In addition, the chaotic experiments in literature make no uniform comparison, which blurs the progress. This work benchmarks the 0Shot-TC problem by providing unified datasets, standardized evaluations, and state-of-the-art baselines. Our contributions include: i) The datasets we provide facilitate studying 0Shot-TC relative to conceptually different and diverse aspects: the ``topic'' aspect includes ``sports'' and ``politics'' as labels; the ``emotion'' aspect includes ``joy'' and ``anger''; the ``situation'' aspect includes ``medical assistance'' and ``water shortage''. ii) We extend the existing evaluation setup (label-partially-unseen) -- given a dataset, train on some labels, test on all labels -- to include a more challenging yet realistic evaluation label-fully-unseen 0Shot-TC (Chang et al., 2008), aiming at classifying text snippets without seeing task specific training data at all. iii) We unify the 0Shot-TC of diverse aspects within a textual entailment formulation and study it this way. Code & Data: https://github.com/yinwenpeng/BenchmarkingZeroShot

Motivation & Objective

• Benchmark 0shot-tc beyond topic categorization to cover diverse aspects (topic, emotion, situation).
• Provide standardized datasets with train/dev/test splits and seen/unseen class separations.
• Extend evaluation to label-partially-unseen and label-fully-unseen settings.
• Propose a unified textual entailment formulation for 0shot-tc to enable open-label generalization.
• Release datasets and code to facilitate fair comparisons.

Proposed method

• Create datasets for three aspects: topic detection, emotion detection, and situation frame detection, with train/dev/test splits and seen/unseen class partitions.
• Introduce label-partially-unseen (standard 0shot-tc) and label-fully-unseen (task-agnostic, no task-specific training) evaluations.
• Formulate 0shot-tc as a textual entailment problem by converting each label into a hypothesis for a given aspect.
• Use BERT-based entailment models (trained on MNLI, FEVER, RTE) to judge text-hypothesis entailment for candidate labels.
• Generate hypotheses for labels using either label names, or label definitions from WordNet, and evaluate single-label and multi-label setups.
• Explore an ensemble approach combining multiple entailment models and hypothesis generation methods.

Experimental results

Research questions

• RQ1 Can standardized datasets and evaluation protocols enable fair comparisons across 0shot-tc methods?
• RQ2 Does framing 0shot-tc as textual entailment enable effective zero-shot classification across diverse aspects without task-specific training?
• RQ3 How do different hypothesis generation strategies (word vs. definition vs. combination) affect 0shot-tc performance?
• RQ4 What is the impact of using pretrained entailment models (MNLI, FEVER, RTE) on label-partially-unseen versus label-fully-unseen settings?
• RQ5 Is an ensemble of entailment models and hypotheses robust across topic, emotion, and situation tasks?

Key findings

• Entailment-based 0shot-tc approaches achieve competitive results for seen labels and substantially improve unseen-label performance across topics, emotions, and situations.
• MNLI-based entailment generally performs strongest in label-partially-unseen settings, with variations across tasks.
• In label-fully-unseen settings, RTE-based models often outperform MNLI and FEVER in some aspects, illustrating dataset-collection effects.
• Word- and definition-based hypothesis generation have task-dependent effects; combining them often yields best results in ensembles.
• A Wikipedia-based pretraining baseline helps topic detection but is less effective for emotion and situation tasks.