[Paper Review] Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules

Daniel E. Ho, Eric Liang+3 more|arXiv (Cornell University)|May 14, 2019

Advanced Neural Network Applications47 references176 citations

TL;DR

The paper introduces Population-Based Augmentation (PBA), which learns nonstationary augmentation schedules via Population Based Training to match AutoAugment’s performance with far less compute. PBA achieves competitive results on CIFAR-10/100 and SVHN with about 1000x reduction in GPU hours.

ABSTRACT

A key challenge in leveraging data augmentation for neural network training is choosing an effective augmentation policy from a large search space of candidate operations. Properly chosen augmentation policies can lead to significant generalization improvements; however, state-of-the-art approaches such as AutoAugment are computationally infeasible to run for the ordinary user. In this paper, we introduce a new data augmentation algorithm, Population Based Augmentation (PBA), which generates nonstationary augmentation policy schedules instead of a fixed augmentation policy. We show that PBA can match the performance of AutoAugment on CIFAR-10, CIFAR-100, and SVHN, with three orders of magnitude less overall compute. On CIFAR-10 we achieve a mean test error of 1.46%, which is a slight improvement upon the current state-of-the-art. The code for PBA is open source and is available at https://github.com/arcelien/pba.

Motivation & Objective

• Motivate data augmentation as a regularizer and identify the challenge of selecting effective augmentation policies from large search spaces.
• Propose a lightweight, schedule-based augmentation search method to improve generalization.
• Reduce computational cost compared to prior AutoAugment approaches while maintaining or improving accuracy.
• Demonstrate PBA's effectiveness across CIFAR-10, CIFAR-100, and SVHN and provide open-source code for reproducibility.

Proposed method

• Formulate augmentation policy search as a hyperparameter schedule learning problem.
• Use Population Based Training (PBT) to jointly optimize augmentation parameters and model training, yielding a time-varying augmentation schedule f(x,t).
• Adopt an augmentation policy template with 15 operations, each having two hyperparameters (probability and magnitude) and two copies of each operation.
• Define a sequence-of-operations application with a limit of 0–2 operations per batch and perturb hyperparameters via a structured explore function.
• Compare against fixed policies (AutoAugment style) and random baselines to assess the value of scheduling versus stationary augmentation.

Experimental results

Research questions

• RQ1 Does learning a nonstationary augmentation schedule yield better generalization than a fixed augmentation policy?
• RQ2 How does PBA performance and efficiency compare to AutoAugment and random baselines across CIFAR-10, CIFAR-100, and SVHN?
• RQ3 What is the impact of the augmentation schedule on computational cost and scalability?
• RQ4 How sensitive is PBA to hyperparameters of the search procedure and schedule length?

Key findings

• PBA matches AutoAugment performance on CIFAR-10, CIFAR-100, and SVHN with roughly 1000x less GPU compute.
• On CIFAR-10, PBA achieves a mean test error of 1.46%, representing a slight improvement over the state-of-the-art at the time.
• PBA requires significantly fewer GPU hours (about 5 hours on a Titan XP) to learn an augmentation schedule, versus thousands of GPU hours for AutoAugment.
• Learning a schedule significantly outperforms fixed policy or stationary distributions, with fixed policies degrading accuracy by about 10% on several models.
• PBA demonstrates robustness across datasets and model families, with competitive results on CIFAR-10/100 and SVHN using learned schedules.