[Paper Review] GANSynth: Adversarial Neural Audio Synthesis

Jesse Engel, Kumar Krishna Agrawal+4 more|arXiv (Cornell University)|Feb 23, 2019

Generative Adversarial Networks and Image Synthesis37 references239 citations

TL;DR

GANSynth demonstrates high-fidelity, locally-coherent audio synthesis by generating log-magnitude spectrograms and instantaneous frequency with GANs, conditioned on pitch, achieving faster generation and competitive perceptual quality against WaveNet on NSynth.

ABSTRACT

Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure at the expense of global latent structure and slow iterative sampling, while Generative Adversarial Networks (GANs), have global latent conditioning and efficient parallel sampling, but struggle to generate locally-coherent audio waveforms. Herein, we demonstrate that GANs can in fact generate high-fidelity and locally-coherent audio by modeling log magnitudes and instantaneous frequencies with sufficient frequency resolution in the spectral domain. Through extensive empirical investigations on the NSynth dataset, we demonstrate that GANs are able to outperform strong WaveNet baselines on automated and human evaluation metrics, and efficiently generate audio several orders of magnitude faster than their autoregressive counterparts.

Motivation & Objective

• Motivate efficient neural audio synthesis that maintains global structure and local waveform coherence.
• Investigate whether GANs can outperform autoregressive models on audio when using spectral-domain representations.
• Assess the impact of different representations (log-magnitude, phase, instantaneous frequency, mel scale) on audio quality and coherence.
• Demonstrate global conditioning for timbre-pitch interpolation and assess generation speed benefits over autoregressive baselines.

Proposed method

• Represent audio as STFT-based spectrograms with magnitude and phase channels or their derivatives (instantaneous frequency).
• Train GANs with progressive growing and gradient penalty, conditioning on pitch as a one-hot vector plus an auxiliary pitch classifier loss.
• Compare representations: log-magnitude with phase, instantaneous frequency, phase, and high-frequency-resolution (+H) variants, including mel-scaled variants (IF-Mel).
• Use an auxiliary classifier GAN setup to encourage pitch-consistent generation.
• Benchmark against WaveGAN and WaveNet baselines on NSynth, using human and automated metrics (NDB, FID, IS, PA, PE).
• Evaluate generation speed and demonstrate parallel, non-autoregressive synthesis.

Experimental results

Research questions

• RQ1 Can GANs synthesize audio with both high perceptual quality and local waveform coherence when trained on spectral representations?
• RQ2 Does incorporating log-m magnitude, instantaneous frequency, and phase representations improve coherence versus direct waveform generation?
• RQ3 How does pitch conditioning influence timbre consistency and perceptual interpolation in GAN-based audio generation?
• RQ4 What are the relative improvements in audio quality and generation speed compared to autoregressive WaveNet and WaveGAN baselines?

Key findings

• GANs trained on log-magnitude spectra with phase or instantaneous frequency can produce more coherent waveforms than direct waveform generation.
• Estimating instantaneous frequency (IF) yields more coherent audio than estimating phase alone.
• Increasing spectral resolution and using mel-scale representations helps separate harmonics and improves quality.
• On NSynth, GANs outperform a strong WaveNet baseline in both automated metrics and human evaluations, while enabling generation orders of magnitude faster than autoregressive models.
• Global conditioning on latent vectors and pitch enables perceptually smooth timbre interpolation and consistent timbre identity across pitch.