[Paper Review] A Bayesian Data Augmentation Approach for Learning Deep Models

Toan Tran, Trung Pham+3 more|arXiv (Cornell University)|Oct 29, 2017

Generative Adversarial Networks and Image Synthesis25 references117 citations

TL;DR

This paper introduces a Bayesian data augmentation framework (GMCEM) that jointly learns a data generator and a classifier for deep models, improving accuracy over standard PMDA and AC-GAN baselines on MNIST, CIFAR-10, and CIFAR-100."

ABSTRACT

Data augmentation is an essential part of the training process applied to deep learning models. The motivation is that a robust training process for deep learning models depends on large annotated datasets, which are expensive to be acquired, stored and processed. Therefore a reasonable alternative is to be able to automatically generate new annotated training samples using a process known as data augmentation. The dominant data augmentation approach in the field assumes that new training samples can be obtained via random geometric or appearance transformations applied to annotated training samples, but this is a strong assumption because it is unclear if this is a reliable generative model for producing new training samples. In this paper, we provide a novel Bayesian formulation to data augmentation, where new annotated training points are treated as missing variables and generated based on the distribution learned from the training set. For learning, we introduce a theoretically sound algorithm --- generalised Monte Carlo expectation maximisation, and demonstrate one possible implementation via an extension of the Generative Adversarial Network (GAN). Classification results on MNIST, CIFAR-10 and CIFAR-100 show the better performance of our proposed method compared to the current dominant data augmentation approach mentioned above --- the results also show that our approach produces better classification results than similar GAN models.

Motivation & Objective

• Motivate robust deep model training when annotated data are limited and expensive.
• Propose a Bayesian data augmentation framework that treats synthetic samples as latent variables drawn from a learned distribution.
• Jointly learn a generator and a classifier within a GMCEM optimization to augment data during training.
• Demonstrate improved classification performance over standard data augmentation and AC-GAN baselines on MNIST, CIFAR-10, and CIFAR-100.

Proposed method

• Introduce latent-variable data augmentation where synthetic samples (z = (ta, xa)) are drawn from a distribution learned from the training data.
• Develop Generalized Monte Carlo EM (GMCEM) to iteratively augment data and update deep model parameters.
• Extend GAN-based generation with a modular architecture comprising a classifier C, an authenticator A, and a generator G, enabling flexible integration with various discriminative and generative models.
• Provide an SGD-based optimization within the GMCEM framework to ensure practical convergence.
• Relate the training losses to the GMCEM objective via lC and lA, linking to the probabilistic formulation (equations 12–19).
• Evaluate on MNIST, CIFAR-10, and CIFAR-100 using ResNet variants, comparing against PMDA and AC-GAN baselines.

Experimental results

Research questions

• RQ1 Does the proposed GMCEM-based Bayesian data augmentation improve classification accuracy over traditional PMDA on standard benchmarks?
• RQ2 How does the Bayesian data augmentation framework compare to AC-GAN and other GAN variants in producing useful augmentation for deep classifiers?
• RQ3 Can the generator/classifier pair be effectively trained jointly within the GMCEM formulation to adapt augmentation during learning?
• RQ4 What is the impact of varying the amount of augmented data (M) on performance across datasets?

Key findings

• The Bayesian DA method improves classification accuracy over PMDA across MNIST, CIFAR-10, and CIFAR-100.
• The method outperforms AC-GAN when using ResNet-based classifiers on the benchmark datasets.
• Increasing the amount of synthesized data (M) generally leads to higher accuracy.
• The method produces reasonably realistic synthetic digits and objects, illustrating practical data augmentation quality.
• Training time analysis shows the Bayesian approach surpasses PMDA after several hours of training on CIFAR-100 and MNIST with suitable architectures.