[论文解读] Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

Ryuichi Yamamoto, Eunwoo Song+1 more|arXiv (Cornell University)|Oct 25, 2019

Speech and Audio Processing参考文献 28被引用 48

一句话总结

Parallel WaveGAN 在不使用蒸馏的设置下，训练非自回归的 WaveNet，使用多分辨率 STFT 与对抗损失，达到 24 kHz 语音，单GPU 实时性为 28.68 倍，参数量为 1.44M，MOS 与基于蒸馏的方法相当。

ABSTRACT

We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.

研究动机与目标

• 推动在不进行密度蒸馏的情况下实现快速且高保真度的波形生成。
• 开发一个简单的训练流水线，结合对抗损失与多分辨率 STFT 损失。
• 使用紧凑模型实现实时或超实时波形生成。
• 在基于 Transformer 的 TTS 框架中展示其有效性。

提出的方法

• 将以声学特征为条件的非自回归 WaveNet 作为生成器。
• 结合波形域对抗损失与多分辨率 STFT 辅助损失进行训练。
• 采用最小平方 GAN 形式以提高训练稳定性。
• 使用不同 FFT 大小、窗口大小和帧移的多分辨率 STFT 损失，以捕捉时频特性。
• 联合优化 L_G = L_aux + lambda_adv * L_adv 以进行生成器训练。
• 与自回归 WaveNet 和 ClariNet 基线进行比较，并在 TTS 设置中使用 MOS 进行评估。

实验结果

研究问题

• RQ1 在不使用蒸馏的 GAN 基声码器是否能在感知质量上与基于蒸馏的系统竞争？
• RQ2 多分辨率 STFT 损失是否改善并行波形生成器对时频语音特征的学习？
• RQ3 在保持高保真度的同时，训练过程是否比两阶段的教师-学生框架更简单更快？
• RQ4 Parallel WaveGAN 在基于 Transformer 的 TTS 框架中作为声码器的表现如何？

主要发现

• Parallel WaveGAN 在单个 V100 GPU 上以 28.68 倍实时速率实现 24 kHz 语音生成，参数量为 1.44M。
• MOS of 4.06 for Parallel WaveGAN (STFT loss only) versus 4.21 for ClariNet with STFT losses in a similar setup.
• Training time for Parallel WaveGAN is 2.8 days, faster than WaveNet (7.4d) and ClariNet (12.7d).
• When integrated with a Transformer TTS model, Parallel WaveGAN reaches MOS of 4.16, competitive with ClariNet-GAN (4.14) and ClariNet (4.00).
• Multi-resolution STFT loss outperforms single-resolution STFT loss and autoregressive WaveNet in perceptual quality.
• Adversarial loss provides robustness benefits in Transformer-based TTS even if benefits are less clear in standalone analysis/synthesis.