[Paper Review] MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Kundan Kumar, Rithesh Kumar+7 more|arXiv (Cornell University)|Oct 8, 2019

Speech and Audio Processing598 citations

TL;DR

MelGAN trains a lightweight, non-autoregressive GAN to invert mel-spectrograms into raw audio with high quality and fast inference, generalizing across speakers and domains.

ABSTRACT

Previous works (Donahue et al., 2018a; Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks.

Motivation & Objective

• Motivate the challenge of generating coherent raw audio with GANs.
• Propose a non-autoregressive, fully convolutional Generator for mel-spectrogram inversion.
• Introduce a multi-scale, window-based Discriminator with hinge loss and feature matching.
• Show efficiency: fewer parameters and fast CPU/GPU inference compared to baselines.
• Demonstrate generalization to unseen speakers and applications in speech, music translation, and unconditional synthesis.

Proposed method

• Generator: fully convolutional upsampling from mel-spectrogram to waveform with dilated residual blocks and careful upsampling to avoid checkerboard artifacts.
• Normalization: weight normalization in all generator layers to stabilize training; avoid instance or spectral normalization.
• Discriminator: multi-scale window-based discriminators operating at raw, 1/2, and 1/4 audio scales with large receptive fields.
• Training objective: hinge loss GAN objective plus feature matching loss (LFM) across discriminator layers; generator optimizes sum of adversarial loss and weighted feature matching (lambda=10).
• Architecture considerations: inductive bias for long-range temporal dependencies via dilated convolutions; control of checkerboard artifacts by kernel/stride choices; no global noise vector input.
• Evaluation: MOS-based listening tests for mel-spectrogram inversion, end-to-end TTS, and non-autoregressive extensions in music translation and VQ-VAE contexts.

Experimental results

Research questions

• RQ1 Can a non-autoregressive, fully convolutional GAN reliably synthesize high-quality raw audio from mel-spectrograms?
• RQ2 Does a multi-scale, window-based discriminator improve audible fidelity and coherence across audio patches?
• RQ3 How does MelGAN compare to autoregressive vocoders and other non-autoregressive methods in MOS?
• RQ4 Can MelGAN generalize to unseen speakers and be integrated into end-to-end TTS and music translation pipelines?

Key findings

• MelGAN achieves competitive MOS scores compared with WaveGlow and WaveNet on mel-spectrogram inversion and end-to-end TTS.
• The model is lightweight (4.26M parameters) and achieves 50-2600 kHz inference speeds across CPU/GPU, outperforming baselines in speed.
• A multi-scale, window-based discriminator and feature matching are crucial for audio quality; removing these components degrades MOS.
• MelGAN generalizes to unseen speakers when trained on multi-speaker data, indicating speaker-invariant mel-to-waveform mappings.
• End-to-end TTS with MelGAN (Text2mel + MelGAN) yields MOS comparable to, or better than, Griffin-Lim and competitive with WaveGlow-based baselines; original high-quality references remain superior.
• MelGAN can replace autoregressive decoders in music translation and VQ-VAE contexts with substantial speedups (e.g., 1s of audio in ~0.16s on GPU).