[Paper Review] Low Resource Text Classification with ULMFit and Backtranslation

Sam Shleifer|arXiv (Cornell University)|Mar 21, 2019

Topic Modeling10 references43 citations

TL;DR

The paper shows that backtranslation substantially improves ULMFit-based text classification in low-resource settings on IMDB, while token perturbations do not; in full data, augmentation has limited benefit, but test-time augmentation and ensembling offer small gains.

ABSTRACT

In computer vision, virtually every state-of-the-art deep learning system is trained with data augmentation. In text classification, however, data augmentation is less widely practiced because it must be performed before training and risks introducing label noise. We augment the IMDB movie reviews dataset with examples generated by two families of techniques: random token perturbations introduced by Wei and Zou [2019] and backtranslation -- translating to a second language then back to English. In low resource environments, backtranslation generates significant improvement on top of the state of-the-art ULMFit model. A ULMFit model pretrained on wikitext103 and then fine-tuned on only 50 IMDB examples and 500 synthetic examples generated by backtranslation achieves 80.6% accuracy, an 8.1% improvement over the augmentation-free baseline with only 9 minutes of additional training time. Random token perturbations do not yield any improvements but incur equivalent computational cost. The benefits of training with backtranslated examples decreases with the size of the available training data. On the full dataset, neither augmentation technique improves upon ULMFit's state of the art performance. We address this by using backtranslations as a form of test time augmentation as well as ensembling ULMFit with other models, and achieve small improvements.

Motivation & Objective

• Motivate robust text classification under low labeled-data conditions.
• Evaluate data augmentation strategies (backtranslation and token perturbations) for NLP in low-resource regimes.
• Assess how backtranslation interacts with pretrained language models like ULMFit.
• Explore test-time augmentation and ensembling to salvage augmentation gains on full datasets.

Proposed method

• Use ULMFit architecture with three-stage training (pretraining on wikitext-103, in-domain finetuning on IMDB, and classifier fine-tuning).
• Augment training data with backtranslation across multiple languages and with token perturbations following Wei and Zou (2019).
• Compare augmentation effects across varying training set sizes (e.g., 50 vs 1000 examples) in low-resource settings.
• Experiment with backtranslation as test-time augmentation (TTA) and with ensembling with other models.
• Optionally discuss Virtual Adversarial Training (VAT) and its impact in ablations.
• Report replicated results versus originally published metrics on full data.

Experimental results

Research questions

• RQ1 Does backtranslation improve ULMFit-based sentiment classification in low-resource settings?
• RQ2 Do token perturbation techniques provide gains over augmentation-free baselines when using ULMFit?
• RQ3 How do augmentation gains scale with training data size, and do they persist on full datasets?
• RQ4 Can backtranslation be leveraged as test-time augmentation and in ensembling to yield additional improvements on full data?

Key findings

• Backtranslation yields significant improvements over augmentation-free ULMFit in low-resource settings, especially with very few labeled examples.
• Token perturbation methods provide little to no improvement over the baseline in low-resource settings.
• As data size grows (toward full IMDB dataset), augmentation gains diminish and can disappear.
• Using backtranslated examples for test-time augmentation (TTA) and ensembling yields small gains on the full dataset.
• Pretraining the language model dominates augmentation gains; augmentation benefits are smaller when strong pretrained representations are available.