[Paper Review] SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

Soroush Mehri, Kundan Kumar+6 more|arXiv (Cornell University)|Dec 22, 2016

Music and Audio Processing362 citations

TL;DR

SampleRNN introduces a hierarchical, multi-rate RNN framework to model raw audio waveforms end-to-end, outperforming baselines in unconditional audio generation and showing human-preferred samples.

ABSTRACT

In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time. We show that our model, which profits from combining memory-less modules, namely autoregressive multilayer perceptrons, and stateful recurrent neural networks in a hierarchical structure is able to capture underlying sources of variations in the temporal sequences over very long time spans, on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.

Motivation & Objective

• Motivate unconditional generation of raw audio without hand-crafted features.
• Develop a multi-tier RNN architecture that captures long-range temporal structure.
• Evaluate performance across diverse audio domains (speech, vocal sounds, music).
• Compare against autoregressive baselines and WaveNet-like models.
• Demonstrate human preference for samples from the proposed model.

Proposed method

• Model the probability of an audio sequence as a product of conditionals using an autoregressive framework.
• Construct a hierarchy of frame-level RNN modules operating at different temporal resolutions.
• Use a sample-level MLP with a discrete output to model next-sample distribution.
• Upsample higher-tier conditioning vectors to feed lower tiers via perforated upsampling.
• Train end-to-end with truncated backpropagation through time and teacher forcing.
• Quantize audio to 256 levels and (optionally) embed inputs before the sample-level MLP.

Experimental results

Research questions

• RQ1 Can a hierarchical, multi-scale RNN model capture long-range dependencies in raw audio better than single-scale models?
• RQ2 What is the impact of frame-level versus sample-level modeling on unconditional audio generation quality?
• RQ3 How does SampleRNN compare to WaveNet and traditional RNN baselines on diverse audio datasets?
• RQ4 Does discretizing the output distribution (multinomial) improve generation quality over real-valued outputs?
• RQ5 How do memory depth and subsequence length affect training and generation quality?

Key findings

• SampleRNN (3-tier) achieves strong log-likelihood scores across Blizzard, Onomatopoeia, and Music datasets, outperforming RNN and WaveNet baselines on several metrics.
• SampleRNN (2-tier) and (3-tier) surpass the baselines in test NLL on Blizzard, Onomatopoeia, and Music datasets (1.392 vs 1.434/1.480/1.410 for Blizzard/Music comparisons).
• Real-valued output variants underperform compared to discrete (256-bin) outputs, with discrete modeling yielding better likelihoods.
• Memory analysis shows SampleRNN can retain speaker identity over seconds and maintain coherence despite silences.
• Human evaluation preferences strongly favor SampleRNN (3-tier) samples over competing models on Blizzard data, with similar trends on Music.
• Subsequence length up to 512 improves validation NLL, indicating benefits from longer context during TBPTT.