[Paper Review] Data Augmentation using Pre-trained Transformer Models

Varun Kumar, Ashutosh Choudhary+1 more|arXiv (Cornell University)|Mar 4, 2020

Topic ModelingComputer Science26 references140 citations

TL;DR

The paper compares conditioning pre-trained transformer models (BERT, GPT-2, BART) for data augmentation across three NLP classification tasks in low-resource settings, finding seq2seq-based BART generally performs best for augmentation while BENCHMARKs like back-translation are strong baselines.

ABSTRACT

Language model based pre-trained models such as BERT have provided significant gains across different NLP tasks. In this paper, we study different types of transformer based pre-trained models such as auto-regressive models (GPT-2), auto-encoder models (BERT), and seq2seq models (BART) for conditional data augmentation. We show that prepending the class labels to text sequences provides a simple yet effective way to condition the pre-trained models for data augmentation. Additionally, on three classification benchmarks, pre-trained Seq2Seq model outperforms other data augmentation methods in a low-resource setting. Further, we explore how different pre-trained model based data augmentation differs in-terms of data diversity, and how well such methods preserve the class-label information.

Motivation & Objective

• Evaluate three transformer-based data augmentation approaches (AE/BERT, AR/GPT-2, Seq2Seq/BART) for conditioned text generation.
• Assess how label conditioning (prepend vs expand) affects augmentation quality across tasks.
• Compare augmentation methods against baselines (EDA, Backtranslation, CBERT) in low-resource regimes.
• Analyze semantic fidelity and diversity of generated data.
• Provide practical guidelines for choosing pre-trained models for data augmentation.

Proposed method

• Fine-tune BERT, GPT-2, and BART on labeled data with class labels prepended to inputs to condition generation.
• For AE models (BERT), use MLM objective with standard masking; for AR models (GPT-2), generate conditioned text with label prompts and extended context; for Seq2Seq models (BART), use denoising with word or span masking and 40% masking rate.
• Compare two conditioning schemes: prepend (label to input) and expand (label as additional token in vocabulary).
• Generate one synthetic example per training example to form a balanced augmented dataset.
• Train a base BERT classifier on augmented data and evaluate on full test sets.
• Evaluate both intrinsic (semantic fidelity via a detector classifier and type-token diversity) and extrinsic (test-set accuracy) effects.

Experimental results

Research questions

• RQ1 Can pre-trained autoencoder, auto-regressive, and seq2seq models be effectively conditioned on class labels for data augmentation?
• RQ2 Which conditioning scheme (prepend vs expand) yields better downstream performance and generalization across tasks?
• RQ3 How do these augmentation methods compare to strong baselines (EDA, Backtranslation, CBERT) in low-resource settings?
• RQ4 How do generated samples balance semantic fidelity and diversity across models?
• RQ5 What practical guidelines emerge for selecting pre-trained models for data augmentation?

Key findings

• Seq2Seq-based BART generally outperforms other augmentation approaches across SST-2, SNIPS, and TREC in low-data settings.
• Back-translation remains a strong baseline for fidelity, often outperforming other pre-trained methods.
• AE-based methods (BERT) show competitive performance, with simple label prepending yielding robust results.
• AR-based GPT-2 requires additional context (GPT-2 context) to better preserve labels in generated data.
• Diversity and fidelity trade-offs vary by method; EDA offers high lexical diversity but may harm semantics.
• Prepend conditioning is often preferable to expansion due to broader compatibility and faster convergence.