[Paper Review] Data Augmentation in Emotion Classification Using Generative Adversarial Networks

Xinyue Zhu, Yifan Liu+2 more|arXiv (Cornell University)|Nov 2, 2017

Generative Adversarial Networks and Image Synthesis25 references76 citations

TL;DR

The paper proposes CycleGAN-based data augmentation to improve emotion-image classification on imbalanced datasets, achieving 5–10% accuracy gains by generating minority-class samples and enriching the data manifold.

ABSTRACT

It is a difficult task to classify images with multiple class labels using only a small number of labeled examples, especially when the label (class) distribution is imbalanced. Emotion classification is such an example of imbalanced label distribution, because some classes of emotions like \emph{disgusted} are relatively rare comparing to other labels like {\it happy or sad}. In this paper, we propose a data augmentation method using generative adversarial networks (GAN). It can complement and complete the data manifold and find better margins between neighboring classes. Specifically, we design a framework with a CNN model as the classifier and a cycle-consistent adversarial networks (CycleGAN) as the generator. In order to avoid gradient vanishing problem, we employ the least-squared loss as adversarial loss. We also propose several evaluation methods on three benchmark datasets to validate GAN's performance. Empirical results show that we can obtain 5%~10% increase in the classification accuracy after employing the GAN-based data augmentation techniques.

Motivation & Objective

• Address the challenge of imbalanced emotion datasets in facial expression recognition.
• Propose a CycleGAN-based data augmentation framework to generate minority-class images.
• Evaluate the impact of GAN-generated data on classification performance across multiple benchmarks.

Proposed method

• Use a CNN classifier for emotion image recognition.
• Employ CycleGAN with least-squared loss (LSGAN) to generate images between reference and target emotion domains.
• Combine LSGAN losses with cycle-consistency loss to train image translations between domains.
• Add GAN-generated samples to the training set and evaluate on FER2013, SFEW, and JAFFE datasets.
• Use t-SNE to visualize data manifold improvements and analyze margins between classes.

Experimental results

Research questions

• RQ1 Can CycleGAN-based data augmentation improve classification accuracy in imbalanced emotion datasets?
• RQ2 Does combining LSGAN loss with cycle-consistency loss stabilize training and improve sample quality?
• RQ3 Do GAN-generated samples meaningfully complete the data manifold and enlarge margins between emotion classes?

Key findings

• GAN-based augmentation yields improvements in overall accuracy on FER2013 when minority classes are augmented.
• Disgust and sad classes show substantial accuracy gains after augmentation.
• Pre-training on GAN-generated data followed by fine-tuning improves performance on SFEW and JAFFE when adapting from FER2013.
• t-SNE visualizations indicate expanded and more distinct class manifolds after augmentation.
• The approach achieves reported gains of approximately 5–10% in emotion classification tasks.