[Paper Review] Data Augmentation for Enhancing EEG-based Emotion Recognition with Deep Generative Models

Yun Luo, Lizhen Zhu+2 more|arXiv (Cornell University)|Jun 4, 2020

EEG and Brain-Computer Interfaces65 references4 citations

TL;DR

This paper proposes three deep generative models—cWGAN, sVAE, and sWGAN—for data augmentation in EEG-based emotion recognition, using variational autoencoders and generative adversarial networks to generate realistic power spectral density and differential entropy features. The method improves classification accuracy by up to 10.2% on the SEED dataset and 5.4% on the DEAP dataset by augmenting training data with high-quality synthetic samples, with selective generation outperforming full augmentation.

ABSTRACT

The data scarcity problem in emotion recognition from electroencephalography (EEG) leads to difficulty in building an affective model with high accuracy using machine learning algorithms or deep neural networks. Inspired by emerging deep generative models, we propose three methods for augmenting EEG training data to enhance the performance of emotion recognition models. Our proposed methods are based on two deep generative models, variational autoencoder (VAE) and generative adversarial network (GAN), and two data augmentation strategies. For the full usage strategy, all of the generated data are augmented to the training dataset without judging the quality of the generated data, while for partial usage, only high-quality data are selected and appended to the training dataset. These three methods are called conditional Wasserstein GAN (cWGAN), selective VAE (sVAE), and selective WGAN (sWGAN). To evaluate the effectiveness of these methods, we perform a systematic experimental study on two public EEG datasets for emotion recognition, namely, SEED and DEAP. We first generate realistic-like EEG training data in two forms: power spectral density and differential entropy. Then, we augment the original training datasets with a different number of generated realistic-like EEG data. Finally, we train support vector machines and deep neural networks with shortcut layers to build affective models using the original and augmented training datasets. The experimental results demonstrate that the augmented training datasets produced by our methods enhance the performance of EEG-based emotion recognition models and outperform the existing data augmentation methods such as conditional VAE, Gaussian noise, and rotational data augmentation.

Motivation & Objective

• Address the critical challenge of data scarcity in EEG-based emotion recognition, where limited labeled datasets hinder deep learning performance.
• Develop deep generative models that produce realistic, high-quality EEG features to augment existing training data.
• Improve the robustness and accuracy of emotion recognition models by enhancing training data diversity and distribution coverage.
• Investigate the impact of varying numbers of generated samples on model performance, identifying optimal augmentation thresholds.
• Compare the effectiveness of full vs. selective data augmentation strategies using VAE and GAN-based generative models.

Proposed method

• Employ conditional Wasserstein GAN (cWGAN) to generate EEG features conditioned on emotion labels, ensuring class-specific data synthesis.
• Implement selective VAE (sVAE) to generate samples and filter only high-quality outputs based on reconstruction loss or discriminative confidence.
• Use selective WGAN (sWGAN) with a quality discriminator to select only realistic-looking generated EEG features for training augmentation.
• Generate two types of EEG features: power spectral density (PSD) and differential entropy (DE), which are standard representations in EEG analysis.
• Augment original training sets with generated samples using both full (all generated data) and partial (only high-quality data) strategies.
• Train and evaluate emotion recognition models using SVMs and deep neural networks with shortcut connections on both original and augmented datasets.

Experimental results

Research questions

• RQ1 Can deep generative models effectively synthesize realistic EEG features to alleviate data scarcity in emotion recognition?
• RQ2 How does selective data augmentation (only high-quality samples) compare to full augmentation (all generated samples) in improving model accuracy?
• RQ3 What is the optimal number of generated samples to add before performance degradation occurs due to noisy or low-quality data?
• RQ4 How do different generative models—VAE vs. GAN—perform in terms of accuracy, stability, and training time for EEG data augmentation?
• RQ5 Does the inclusion of synthetic EEG data improve generalization and robustness of emotion recognition models across multiple datasets and classifiers?

Key findings

• The proposed methods improved emotion recognition accuracy by up to 10.2% on the SEED dataset and 5.4% on the DEAP dataset compared to baseline models.
• Selective WGAN (sWGAN) achieved the highest classification performance among all methods, outperforming both sVAE and cWGAN.
• Performance decayed when more than 10 times the original training set size of generated samples were added, indicating a threshold beyond which noise degrades model accuracy.
• sVAE showed better training stability than GAN-based methods due to the absence of adversarial training instabilities like mode collapse.
• cWGAN had the fastest convergence speed, as it used all generated samples without quality filtering, but suffered from lower accuracy due to inclusion of low-quality data.
• The generated data improved model generalization by expanding the data manifold and creating better decision margins, especially when high-quality samples were used.