[Paper Review] Towards Zero-Label Language Learning

Zi-Rui Wang, Adams Wei Yu+2 more|arXiv (Cornell University)|Sep 19, 2021

Topic ModelingComputer Science23 references46 citations

TL;DR

The paper introduces Unsupervised Data Generation (UDG), a zero-label learning framework that uses few-shot prompts with a pretrained language model to synthesize labeled training data from unlabeled examples, enabling zero-label training and effective data augmentation that can surpass supervised baselines on SuperGLUE.

ABSTRACT

This paper explores zero-label learning in Natural Language Processing (NLP), whereby no human-annotated data is used anywhere during training and models are trained purely on synthetic data. At the core of our framework is a novel approach for better leveraging the powerful pretrained language models. Specifically, inspired by the recent success of few-shot inference on GPT-3, we present a training data creation procedure named Unsupervised Data Generation (UDG), which leverages few-shot prompts to synthesize high-quality training data without real human annotations. Our method enables zero-label learning as we train task-specific models solely on the synthetic data, yet we achieve better or comparable results from strong baseline models trained on human-labeled data. Furthermore, when mixed with labeled data, our approach serves as a highly effective data augmentation procedure, achieving new state-of-the-art results on the SuperGLUE benchmark.

Motivation & Objective

• Motivate zero-label learning in NLP to reduce reliance on human-annotated data.
• Develop a data-creation pipeline that leverages pretrained language models to generate labeled examples without supervision.
• Show that training on synthetic data can match or surpass supervised baselines and enhance performance when combined with labeled data.
• Demonstrate the efficacy of UDG across text classification and general language understanding benchmarks (SuperGLUE).

Proposed method

• Propose Unsupervised Data Generation (UDG): generate labeled data by prompting a pretrained language model to produce inputs X given a pseudo-label description Des(y_hat) and a few unlabeled examples, thereby decoupling generation from prediction.
• Use top-k sampling to generate synthetic (x_g, y_hat) pairs without fine-tuning the generator.
• Train task-specific models on the synthetic dataset, optionally with Noisy Label Annealing (NLA) to downweight noisy synthetic examples during training.
• Apply UDG for unsupervised text classification and for SuperGLUE-style language understanding tasks, using T5-based downstream models for fine-tuning on created data.
• Investigate data-size and prompt-example effects, and study data augmentation by combining synthetic data with real labeled data.
• Compare against few-shot inference and supervised baselines to evaluate competitiveness and potential super-human performance on SuperGLUE when augmented with labeled data.

Experimental results

Research questions

• RQ1 Can zero-label learning via synthetic data generation match or surpass supervised baselines on standard NLP benchmarks?
• RQ2 How does synthetic data quality, prompt design, and dataset size affect UDG performance?
• RQ3 Is UDG effective as a data augmentation method when combined with real labeled data, particularly on challenging benchmarks like SuperGLUE?
• RQ4 What is the role of Noisy Label Annealing in mitigating noisy synthetic data during training?

Key findings

• UDG achieves competitive results with strong supervised baselines on text classification benchmarks (IMDb, Yelp, Amazon, DBpedia) in unsupervised settings.
• On SuperGLUE, UDG achieves 78.1 average with unsupervised data, and when combined with labeled data, reaches state-of-the-art scores, including surpassing human performance as of Dec 20, 2020.
• Using T5 or XLNet-based backbones, UDG plus Noisy Label Annealing (NLA) improves performance across tasks, outperforming few-shot inference like GPT-3 in many cases.
• Data generated via few-shot prompts can be scaled for augmentation, yielding improvements particularly on small-data tasks (COPA, WSC) when fused with labeled data.
• A key finding is that UDG can outperform several fully supervised models on IMDb and Yelp-2 in zero-label settings, highlighting the strength of unsupervised data creation for downstream training.