[Paper Review] EnvGAN: Adversarial Synthesis of Environmental Sounds for Data Augmentation

Aswathy Madhu, Suresh Kumaraswamy|arXiv (Cornell University)|Apr 15, 2021

Music and Audio Processing34 references4 citations

TL;DR

This paper introduces EnvGAN, the first Generative Adversarial Network (GAN) for adversarial synthesis of environmental sounds to address data scarcity in Environmental Sound Classification (ESC). By generating diverse, realistic audio samples in the time domain, EnvGAN significantly boosts classifier performance, outperforming state-of-the-art methods on ESC-10, UrbanSound8K, and TUT Urban Acoustic Scenes 2018 datasets with accuracy gains up to 0.965, 0.9897, and 0.73, respectively.

ABSTRACT

The research in Environmental Sound Classification (ESC) has been progressively growing with the emergence of deep learning algorithms. However, data scarcity poses a major hurdle for any huge advance in this domain. Data augmentation offers an excellent solution to this problem. While Generative Adversarial Networks (GANs) have been successful in generating synthetic speech and sounds of musical instruments, they have hardly been applied to the generation of environmental sounds. This paper presents EnvGAN, the first ever application of GANs for the adversarial generation of environmental sounds. Our experiments on three standard ESC datasets illustrate that the EnvGAN can synthesize audio similar to the ones in the datasets. The suggested method of augmentation outshines most of the futuristic techniques for audio augmentation.

Motivation & Objective

• To address data scarcity in Environmental Sound Classification (ESC), a major bottleneck in deep learning applications.
• To develop a novel GAN-based method for synthesizing realistic environmental sounds that preserve semantic content.
• To evaluate the effectiveness of GAN-generated data as an augmentation strategy for improving ESC model generalization and robustness.
• To demonstrate that EnvGAN outperforms traditional and advanced data augmentation techniques in ESC tasks.
• To explore the potential of class-conditional augmentation for further performance gains.

Proposed method

• Proposes a time-domain GAN architecture (EnvGAN) that generates environmental sounds by learning the latent distribution of real audio waveforms.
• Employs a deep convolutional neural network (CNN) as the generator and a discriminative network to distinguish real from generated audio samples.
• Trains the generator and discriminator using a minimax loss function, minimizing the Jensen-Shannon divergence between real and generated data distributions.
• Applies the trained EnvGAN to generate synthetic audio samples that are added to the original training set for data augmentation.
• Uses a high-capacity CNN model as the baseline classifier, fine-tuned on both original and augmented datasets to evaluate performance.
• Employs a validation set to identify class-specific augmentation effects, enabling selective, performance-driven data augmentation.

Experimental results

Research questions

• RQ1 Can a GAN-based approach effectively generate realistic environmental sounds that preserve semantic labels?
• RQ2 Does GAN-based data augmentation improve the generalization and accuracy of deep learning models in environmental sound classification?
• RQ3 How does EnvGAN-generated data compare to traditional and state-of-the-art audio augmentation techniques in terms of performance on benchmark datasets?
• RQ4 Which classes benefit most or least from GAN-based augmentation, and can class-conditional augmentation further improve model performance?
• RQ5 Can EnvGAN effectively handle class-imbalanced datasets like UrbanSound8K through synthetic sample generation?

Key findings

• EnvGAN successfully generates environmental sounds that are perceptually similar to real samples from the ESC-10, UrbanSound8K, and TUT Urban Acoustic Scenes 2018 datasets.
• The proposed method achieves a test accuracy of 0.965 on ESC-10, surpassing the previous state-of-the-art model by 0.0155.
• On UrbanSound8K, the model reaches 0.9897 accuracy, significantly outperforming the next best method by 0.0145.
• For the TUT Urban Acoustic Scenes 2018 dataset, the model achieves 0.73 accuracy, demonstrating strong performance on a more complex and diverse dataset.
• Augmentation reduces confusion between certain classes (e.g., sea_waves and rain) but increases confusion between others (e.g., sea_waves and person_sneeze), indicating class-specific effects.
• The model shows the most significant performance gain on class-imbalanced datasets, confirming EnvGAN’s effectiveness in mitigating data imbalance.