[Paper Review] WaveNet: A Generative Model for Raw Audio

Aäron van den Oord, Sander Dieleman+7 more|arXiv (Cornell University)|Sep 12, 2016

Music and Audio Processing3,585 citations

TL;DR

WaveNet is a fully probabilistic autoregressive model that generates raw audio waveforms using dilated causal convolutions, achieving state-of-the-art naturalness in TTS and promising results in music and speech recognition.

ABSTRACT

This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-of-the-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.

Motivation & Objective

• Demonstrate a neural network that generates raw audio waveforms end-to-end.
• Develop dilated causal convolution architectures to capture long-range temporal dependencies in audio.
• Show that conditioning on speaker identity enables multi-speaker generation with a single model.
• Evaluate WaveNet on text-to-speech, multi-speaker speech generation, and music modeling; explore discriminative uses like speech recognition.

Proposed method

• Model the joint waveform probability as a product of conditionals p(x_t | x_1,...,x_{t-1}).
• Use causal convolutions with dilation to achieve very large receptive fields without recurrent nets.
• Quantize audio via μ-law and model conditional next-sample distribution with a softmax over 256 values.
• Incorporate gated activation units and residual/skip connections for deep architectures.
• Support global and local conditioning to steer generation (e.g., speaker identity, linguistic features).
• Optionally stack context modules to manage long-range dependencies and heterogeneous timescales.

Experimental results

Research questions

• RQ1 Can a fully probabilistic autoregressive model generate high-fidelity raw audio directly at the waveform level?
• RQ2 Do dilated causal convolutions provide sufficient receptive fields to model long-range audio dependencies without RNNs?
• RQ3 Can a single WaveNet model capture multiple speakers, and can conditioning control voice characteristics?
• RQ4 How does WaveNet perform on text-to-speech compared to traditional baselines, and can it model music and support speech recognition tasks?

Key findings

• WaveNet achieves state-of-the-art subjective naturalness in text-to-speech across English and Mandarin.
• A single WaveNet can model multiple speakers when conditioned on speaker identity.
• Dilated causal convolutions provide exponentially growing receptive fields enabling long-range dependency modeling.
• WaveNet conditioned on linguistic features and F0 can outperform statistical parametric and concatenative baselines in MOS tests.
• In TIMIT, WaveNet with an auxiliary loss for frame classification achieves 18.8% phone error rate, competitive with raw-audio discriminative models.
• Conditional WaveNets can generate musical fragments with perceptual quality, and global/local conditioning supports control over output attributes.