[Paper Review] Neural Data Augmentation via Example Extrapolation

Kenton Lee, Kelvin Guu+3 more|arXiv (Cornell University)|Feb 2, 2021

Topic Modeling32 references37 citations

TL;DR

Ex2 trains a neural example extrapolator to synthesize new labeled data from few exemplars within underrepresented data slices, boosting few-shot performance across CLINC150, SNIPS, and FewRel benchmarks.

ABSTRACT

In many applications of machine learning, certain categories of examples may be underrepresented in the training data, causing systems to underperform on such "few-shot" cases at test time. A common remedy is to perform data augmentation, such as by duplicating underrepresented examples, or heuristically synthesizing new examples. But these remedies often fail to cover the full diversity and complexity of real examples. We propose a data augmentation approach that performs neural Example Extrapolation (Ex2). Given a handful of exemplars sampled from some distribution, Ex2 synthesizes new examples that also belong to the same distribution. The Ex2 model is learned by simulating the example generation procedure on data-rich slices of the data, and it is applied to underrepresented, few-shot slices. We apply Ex2 to a range of language understanding tasks and significantly improve over state-of-the-art methods on multiple few-shot learning benchmarks, including for relation extraction (FewRel) and intent classification + slot filling (SNIPS).

Motivation & Objective

• Motivate the problem of underrepresented slices causing weak few-shot performance in NLP tasks.
• Propose Ex2, a neural example extrapolator that generates new examples from a small set of exemplars from the same slice.
• Formalize data slicing and the Ex2 training objective to model p(e|s) from e1:K exemplars.
• Demonstrate empirical gains of Ex2 data augmentation on text classification, intent-slot tasks, and relation extraction.

Proposed method

• Organize training data into slices via user-defined slice functions and designate few-shot vs many-shot slices.
• Train a sequence-to-sequence extrapolator (Ex2) using data-rich slices by maximizing the likelihood of a held-out example given K exemplars from the same slice.
• Represent exemplars and outputs as anonymized text to prevent leakage of slice identities and encourage genuine distribution learning.
• Use the trained Ex2 to generate synthetic labeled examples for underrepresented slices and augment the downstream model’s training set.
• Cast downstream models as T5-based seq2seq learners and evaluate on both overall and few-shot performance across tasks.

Experimental results

Research questions

• RQ1 Can Ex2 effectively learn the full distribution of a data slice given only a few exemplars from that slice?
• RQ2 Does anonymizing slice identities help Ex2 generalize beyond memorization of slice labels?
• RQ3 To what extent does the number of exemplars K affect Ex2’s ability to synthesize representative new data?
• RQ4 Is Ex2 data augmentation beneficial across diverse NLP tasks with different output spaces (classification, slot filling, relation extraction)?

Key findings

• Ex2 yields strong few-shot gains across tasks, e.g., CLINC150: overall 97.4% accuracy, macro F1 96.1, few-shot accuracy 95.6%, few-shot macro F1 80.4%.
• In SNIPS, Ex2 attains intent accuracy 97.8% overall and 93.5% few-shot; slot F1 rises to 75.3 on the few-shot slices.
• For FewRel-Open, Ex2 achieves 78.0% overall accuracy and 70.7% few-shot accuracy, outperforming baselines and previous state-of-the-art approaches in this setting.
• Ablations show that more exemplars (higher K) and anonymization of slice labels improve few-shot performance, while world-knowledge pretraining (T5) is crucial for effective Ex2.
• Pre-trained Ex2 models (e.g., T5-XL) significantly outperform randomly initialized ones, underscoring the importance of large-scale pretraining for world knowledge in extrapolation.