[Paper Review] DiffWave: A Versatile Diffusion Model for Audio Synthesis

Zhifeng Kong, Wei Ping+3 more|arXiv (Cornell University)|Sep 21, 2020

Music and Audio Processing51 references121 citations

TL;DR

DiffWave is a non-autoregressive diffusion model that synthesizes raw audio, matching WaveNet vocoder quality with much faster synthesis and excelling in unconditional and class-conditional generation.

ABSTRACT

In this work, we propose DiffWave, a versatile diffusion probabilistic model for conditional and unconditional waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audios in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality (MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.

Motivation & Objective

• Advance high-fidelity raw-audio synthesis with a non-autoregressive diffusion model.
• Enable both conditional (mel-spectrogram and global labels) and unconditional waveform generation.
• Achieve competitive speech quality with a small footprint and efficient training using ELBO-based objectives.

Proposed method

• Use a diffusion probabilistic framework to map from white noise to audio via a Markov reverse process.
• Adopt the closed-form ELBO training viaHo et al. parameterization with ϵθ denoiser and fixed σθ
• Implement DiffWave with a feed-forward bidirectional dilated convolution architecture (Bi-DilConv) inspired by WaveNet but non-autoregressive.
• Incorporate diffusion-step embeddings to condition ϵθ on the current diffusion step.
• Support local (mel spectrogram) and global (speaker/word) conditioning, plus unconditional generation by leveraging large receptive fields across diffusion steps.
• Provide fast sampling by collapsing the reverse process to Tinfer steps with a designed variance schedule.

Experimental results

Research questions

• RQ1 Can DiffWave achieve high-fidelity neural vocoding conditioned on mel spectrograms while offering substantially faster synthesis than autoregressive vocoders?
• RQ2 How does DiffWave perform on unconditional waveform generation in terms of audio quality and sample diversity compared to autoregressive and GAN-based models?
• RQ3 Does DiffWave deliver competitive or superior results in class-conditional waveform generation relative to baseline autoregressive models?
• RQ4 What are the trade-offs between model footprint, speed, and quality across different diffusion-step counts and model sizes?

Key findings

• DiffWave matches WaveNet vocoder quality (MOS: 4.44 vs. 4.43) with orders-of-magnitude faster synthesis.
• A small DiffWave model (2.64M params) achieves MOS 4.37 on 22.05 kHz speech and runs >5× faster than real-time on V100 without engineered kernels.
• DiffWave outperforms WaveNet and WaveGAN in unconditional generation in terms of audio quality and sample diversity across multiple automatic and human evaluations.
• In unconditional generation, DiffWave achieves MOS 3.39, outperforming WaveNet (MOS 1.43) and WaveGAN (MOS 2.03).
• In class-conditional generation, DiffWave achieves MOS 3.50 (vs WaveNet 1.58), and a deep version reaches MOS 3.44.
• DiffWave demonstrates strong unconditional and conditional generation with competitive quality while maintaining a smaller footprint than many flow-based vocoders.