Skip to main content
QUICK REVIEW

[Paper Review] Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

Ryuichi Yamamoto, Eunwoo Song|arXiv (Cornell University)|Oct 25, 2019
Speech and Audio ProcessingComputer Science28 references48 citations
TL;DR

Parallel WaveGAN trains a non-autoregressive WaveNet with multi-resolution STFT and adversarial losses in a distillation-free setup, achieving 24 kHz speech at 28.68x real-time on a single GPU with 1.44M params and MOS comparable to distillation-based methods.

ABSTRACT

We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.

Motivation & Objective

  • Motivate fast, high-fidelity waveform generation without density distillation.
  • Develop a simple training pipeline that combines adversarial and multi-resolution STFT losses.
  • Achieve real-time or faster-than-real-time waveform generation with a compact model.
  • Demonstrate effectiveness within a Transformer-based TTS framework.

Proposed method

  • Use a non-autoregressive WaveNet conditioned on acoustic features as the generator.
  • Train with a combination of waveform-domain adversarial loss and multi-resolution STFT auxiliary loss.
  • Adopt least-squares GAN formulation for stability in training.
  • Employ multi-resolution STFT losses with different FFT sizes, window sizes, and frame shifts to capture time-frequency characteristics.
  • Jointly optimize L_G = L_aux + lambda_adv * L_adv for generator training.
  • Compare against autoregressive WaveNet and ClariNet baselines, and evaluate with MOS in TTS setups.

Experimental results

Research questions

  • RQ1Can a distillation-free GAN-based vocoder achieve competitive perceptual quality versus distillation-based systems?
  • RQ2Does multi-resolution STFT loss improve learning of time-frequency speech characteristics in a parallel waveform generator?
  • RQ3Is the training process simpler and faster than two-stage teacher-student frameworks while maintaining high fidelity?
  • RQ4How does Parallel WaveGAN perform as a vocoder within a Transformer-based TTS framework?

Key findings

  • Parallel WaveGAN achieves 24 kHz speech generation at 28.68x real-time on a single V100 GPU with 1.44M parameters.
  • MOS of 4.06 for Parallel WaveGAN (STFT loss only) versus 4.21 for ClariNet with STFT losses in a similar setup.
  • Training time for Parallel WaveGAN is 2.8 days, faster than WaveNet (7.4d) and ClariNet (12.7d).
  • When integrated with a Transformer TTS model, Parallel WaveGAN reaches MOS of 4.16, competitive with ClariNet-GAN (4.14) and ClariNet (4.00).
  • Multi-resolution STFT loss outperforms single-resolution STFT loss and autoregressive WaveNet in perceptual quality.
  • Adversarial loss provides robustness benefits in Transformer-based TTS even if benefits are less clear in standalone analysis/synthesis.

Read this paper

Open the full text in the Nubint reader and ask about any passage as you read.

Read this paper on Nubint

This review was generated by AI from the original paper. Please check the paper before citing it.