[Paper Review] Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

Ryuichi Yamamoto, Eunwoo Song+1 more|arXiv (Cornell University)|Oct 25, 2019

Speech and Audio Processing28 references48 citations

TL;DR

Parallel WaveGAN trains a non-autoregressive WaveNet with multi-resolution STFT and adversarial losses in a distillation-free setup, achieving 24 kHz speech at 28.68x real-time on a single GPU with 1.44M params and MOS comparable to distillation-based methods.

ABSTRACT

We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.

Motivation & Objective

• Motivate fast, high-fidelity waveform generation without density distillation.
• Develop a simple training pipeline that combines adversarial and multi-resolution STFT losses.
• Achieve real-time or faster-than-real-time waveform generation with a compact model.
• Demonstrate effectiveness within a Transformer-based TTS framework.

Proposed method

• Use a non-autoregressive WaveNet conditioned on acoustic features as the generator.
• Train with a combination of waveform-domain adversarial loss and multi-resolution STFT auxiliary loss.
• Adopt least-squares GAN formulation for stability in training.
• Employ multi-resolution STFT losses with different FFT sizes, window sizes, and frame shifts to capture time-frequency characteristics.
• Jointly optimize L_G = L_aux + lambda_adv * L_adv for generator training.
• Compare against autoregressive WaveNet and ClariNet baselines, and evaluate with MOS in TTS setups.

Experimental results

Research questions

• RQ1 Can a distillation-free GAN-based vocoder achieve competitive perceptual quality versus distillation-based systems?
• RQ2 Does multi-resolution STFT loss improve learning of time-frequency speech characteristics in a parallel waveform generator?
• RQ3 Is the training process simpler and faster than two-stage teacher-student frameworks while maintaining high fidelity?
• RQ4 How does Parallel WaveGAN perform as a vocoder within a Transformer-based TTS framework?

Key findings

• Parallel WaveGAN achieves 24 kHz speech generation at 28.68x real-time on a single V100 GPU with 1.44M parameters.
• MOS of 4.06 for Parallel WaveGAN (STFT loss only) versus 4.21 for ClariNet with STFT losses in a similar setup.
• Training time for Parallel WaveGAN is 2.8 days, faster than WaveNet (7.4d) and ClariNet (12.7d).
• When integrated with a Transformer TTS model, Parallel WaveGAN reaches MOS of 4.16, competitive with ClariNet-GAN (4.14) and ClariNet (4.00).
• Multi-resolution STFT loss outperforms single-resolution STFT loss and autoregressive WaveNet in perceptual quality.
• Adversarial loss provides robustness benefits in Transformer-based TTS even if benefits are less clear in standalone analysis/synthesis.