[Paper Review] Adversarial Audio Synthesis

Chris Donahue, Julian McAuley+1 more|arXiv (Cornell University)|Feb 12, 2018

Generative Adversarial Networks and Image SynthesisComputer Science94 citations

TL;DR

The paper introduces WaveGAN and SpecGAN for unsupervised raw-audio generation using GANs, demonstrating one-second coherent audio across domains (speech, drums, birds, piano) and evaluating with human judgments and inception-based metrics.

ABSTRACT

Audio signals are sampled at high temporal resolutions, and learning to synthesize audio requires capturing structure across a range of timescales. Generative adversarial networks (GANs) have seen wide success at generating images that are both locally and globally coherent, but they have seen little application to audio generation. In this paper we introduce WaveGAN, a first attempt at applying GANs to unsupervised synthesis of raw-waveform audio. WaveGAN is capable of synthesizing one second slices of audio waveforms with global coherence, suitable for sound effect generation. Our experiments demonstrate that, without labels, WaveGAN learns to produce intelligible words when trained on a small-vocabulary speech dataset, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano. We compare WaveGAN to a method which applies GANs designed for image generation on image-like audio feature representations, finding both approaches to be promising.

Motivation & Objective

• Motivate and explore unsupervised generation of raw audio with GANs instead of relying on spectrograms or autoregressive models.
• Develop WaveGAN to generate one-second raw-waveform audio with a GAN architecture adapted from DCGAN.
• Develop SpecGAN to generate semi-invertible spectrograms and compare with waveform-based generation.
• Evaluate generation quality, diversity, and intelligibility on multiple audio domains including speech, drums, bird vocalizations, and piano.
• Provide practical guidelines and a baseline template for adapting image-generation GANs to audio synthesis.

Proposed method

• Adapt DCGAN to one-dimensional waveform generation (WaveGAN) by using longer 1D filters (length 25) and upsampling by 4, with phase shuffle in the discriminator to mitigate artifacts.
• Introduce SpecGAN that operates on spectrograms with an approximate invertibility path, using Griffin-Lim for waveform recovery.
• Apply WGAN-GP training to stabilize GAN optimization.
• Use phase shuffle to enforce phase invariance in the discriminator (n in {1,2,4}).
• Evaluate using inception score on an audio classifier trained for SC09, along with diversity and train-data proximity metrics, plus human judgments.

Experimental results

Research questions

• RQ1 Can GANs learn global structure in high-dimensional audio data without conditioning labels?
• RQ2 How do waveform-based and spectrogram-based GAN approaches compare for unsupervised audio synthesis across different domains?
• RQ3 What regularization or architectural choices (e.g., phase shuffle) improve audio generation quality and diversity?
• RQ4 Is generated audio intelligible to humans, and how does it fare against real data and autoregressive baselines?

Key findings

• WaveGAN and SpecGAN can generate intelligible spoken digits in an unconditional setting.
• WaveGAN generally yields higher subjective sound quality and speaker diversity than SpecGAN, despite SpecGAN achieving higher inception scores.
• Phase shuffle improves or stabilizes generation for some configurations, notably outperforming certain regularizers like dropout in this setting.
• WaveGAN attains an inception score of 4.7 on SC09; SpecGAN attains 6.03 in the same setting, with real test data scoring 8.01.
• Across domains (drums, birds, piano, TIMIT), WaveGAN produces perceptually coherent audio samples and captures domain-consistent structure; SpecGAN variants show stronger variance capture but can suffer from inversion-related artifacts.