[Paper Review] Zero-shot Text Classification With Generative Language Models

Raul Puri, Bryan Catanzaro|arXiv (Cornell University)|Dec 10, 2019

Topic Modeling22 references74 citations

TL;DR

The paper shows a generative language-model-based approach to zero-shot text classification by framing tasks as natural language multiple-choice questions, enabling adaptation to new tasks without task-specific heads and achieving gains over baselines on six datasets.

ABSTRACT

This work investigates the use of natural language to enable zero-shot model adaptation to new tasks. We use text and metadata from social commenting platforms as a source for a simple pretraining task. We then provide the language model with natural language descriptions of classification tasks as input and train it to generate the correct answer in natural language via a language modeling objective. This allows the model to generalize to new classification tasks without the need for multiple multitask classification heads. We show the zero-shot performance of these generative language models, trained with weak supervision, on six benchmark text classification datasets from the torchtext library. Despite no access to training data, we achieve up to a 45% absolute improvement in classification accuracy over random or majority class baselines. These results show that natural language can serve as simple and powerful descriptors for task adaptation. We believe this points the way to new metalearning strategies for text problems.

Motivation & Objective

• Motivate zero-shot model adaptation for new text classification tasks using natural language descriptors.
• Pretrain a generative language model on a diverse, weakly supervised task to simulate many N-way classification tasks.
• Evaluate zero-shot transfer to six standard text classification benchmarks and analyze the effect of model and data scale.
• Explore how class descriptor quality affects performance and controllability, and discuss limitations and future directions.

Proposed method

• Reformulate text classification as a multiple-choice question answering task with natural language class descriptions.
• Pretrain a GPT-2 model on a large weakly supervised task (OpenWebText with title prediction) to learn to select descriptive labels from a set of natural language options.
• Prepend a selected list of class descriptions to the input document and train the model to generate the correct class text via a language modeling objective.
• Use learned positional embeddings and special tokens to separate question, text, and answer segments, and optimize with next-token prediction loss.
• Evaluate zero-shot performance on six TorchText datasets by providing the full set of classes in natural language and prompting the model to output the correct class.
• Investigate model size and data scale effects, and compare with random, majority, and finetuned baselines.

Experimental results

Research questions

• RQ1 Can a single generative language model adapt to unseen text classification tasks using natural language task descriptions?
• RQ2 How do model size and pretraining data scale influence zero-shot classification accuracy across diverse datasets?
• RQ3 What is the impact of the quality of class descriptors on zero-shot performance and output validity?
• RQ4 To what extent can a unified model avoid task-specific heads while maintaining competitive accuracy?
• RQ5 What are the limitations and potential future directions for zero-shot text classification with generative models?

Key findings

• Increasing model size and pretraining data improves zero-shot performance across tasks.
• On DBPedia, the 355M GPT-2 model achieves up to a 45.2% absolute improvement over random baselines.
• For several datasets (DBPedia, AGNews, Yahoo Answers), performance surpasses random and majority baselines but remains below finetuned or SOTA levels.
• Descriptor quality significantly impacts accuracy; bad descriptors can cause large drops (up to 27% absolute) and more out-of-vocabulary outputs.
• Greedy decoding yields fewer out-of-vocabulary outputs compared to sampling methods like top-k/top-p.
• Rare out-of-vocabulary outputs are typically miscombinations or empty strings, and larger models reduce such errors.