[Paper Review] Unsupervised Data Augmentation

Qizhe Xie, Zihang Dai+3 more|arXiv (Cornell University)|Apr 29, 2019

Topic ModelingComputer Science18 references175 citations

TL;DR

UDA proposes a semi-supervised learning method that applies data augmentation to unlabeled data, enforcing prediction consistency between original and augmented unlabeled examples using realistic noise from state-of-the-art augmentation policies. This approach achieves state-of-the-art results on six NLP and three vision tasks, reducing error rates by over 30% on CIFAR-10 and SVHN even with minimal labeled data.

ABSTRACT

Despite its success, deep learning still needs large labeled datasets to succeed. Data augmentation has shown much promise in alleviating the need for more labeled data, but it so far has mostly been applied in supervised settings and achieved limited gains. In this work, we propose to apply data augmentation to unlabeled data in a semi-supervised learning setting. Our method, named Unsupervised Data Augmentation or UDA, encourages the model predictions to be consistent between an unlabeled example and an augmented unlabeled example. Unlike previous methods that use random noise such as Gaussian noise or dropout noise, UDA has a small twist in that it makes use of harder and more realistic noise generated by state-of-the-art data augmentation methods. This small twist leads to substantial improvements on six language tasks and three vision tasks even when the labeled set is extremely small. For example, on the IMDb text classification dataset, with only 20 labeled examples, UDA outperforms the state-of-the-art model trained on 25,000 labeled examples. On standard semi-supervised learning benchmarks, CIFAR-10 with 4,000 examples and SVHN with 1,000 examples, UDA outperforms all previous approaches and reduces more than $30\%$ of the error rates of state-of-the-art methods: going from 7.66% to 5.27% and from 3.53% to 2.46% respectively. UDA also works well on datasets that have a lot of labeled data. For example, on ImageNet, with 1.3M extra unlabeled data, UDA improves the top-1/top-5 accuracy from 78.28/94.36% to 79.04/94.45% when compared to AutoAugment.

Motivation & Objective

• To reduce reliance on large labeled datasets in deep learning by leveraging unlabeled data through data augmentation.
• To address the limited effectiveness of data augmentation in unsupervised and semi-supervised settings.
• To improve model generalization by enforcing prediction consistency between original and augmented unlabeled examples.
• To explore whether realistic, strong augmentation policies can outperform random noise in semi-supervised training.

Proposed method

• UDA applies data augmentation to unlabeled examples and enforces consistency in model predictions between the original and augmented versions.
• The method uses state-of-the-art data augmentation policies—such as AutoAugment or Cutout—to generate realistic, non-random noise instead of Gaussian or dropout noise.
• Prediction consistency is enforced via a consistency regularization loss that minimizes the KL divergence between predictions of original and augmented unlabeled inputs.
• The model is trained end-to-end using both labeled data (cross-entropy loss) and unlabeled data (consistency loss), enabling semi-supervised learning.
• UDA is applied in a self-training fashion, where predictions on augmented unlabeled data are used to refine the model.
• The approach is domain-agnostic and effective across vision and language tasks, including ImageNet, CIFAR-10, SVHN, and IMDb.

Experimental results

Research questions

• RQ1 Can data augmentation applied to unlabeled data significantly improve model performance in semi-supervised learning?
• RQ2 Does using realistic, strong augmentation policies outperform random noise in consistency regularization?
• RQ3 How effective is UDA when labeled data is extremely limited, such as 20 examples in text classification?
• RQ4 Can UDA achieve state-of-the-art results on standard benchmarks like CIFAR-10 and SVHN with minimal labeled data?
• RQ5 Does UDA scale effectively to large-scale datasets like ImageNet with abundant unlabeled data?

Key findings

• On the IMDb text classification dataset, UDA achieved state-of-the-art performance with only 20 labeled examples, outperforming a model trained on 25,000 labeled examples.
• On CIFAR-10 with 4,000 labeled examples, UDA reduced the error rate from 7.66% to 5.27%, a 31.1% relative reduction.
• On SVHN with 1,000 labeled examples, UDA reduced the error rate from 3.53% to 2.46%, a 30.3% relative reduction.
• On ImageNet, UDA improved top-1 accuracy from 78.28% to 79.04% and top-5 accuracy from 94.36% to 94.45% using 1.3 million extra unlabeled images.
• UDA consistently outperformed all prior methods on all six language and three vision benchmarks tested, demonstrating broad effectiveness.