[Paper Review] Efficient Neural Audio Synthesis

Nal Kalchbrenner, Erich Elsen+8 more|arXiv (Cornell University)|Feb 23, 2018

Music and Audio ProcessingComputer Science20 references75 citations

TL;DR

Introduces WaveRNN, a lightweight recurrent model for 24 kHz 16-bit audio that achieves real-time or faster-than-real-time synthesis with high fidelity, via dual softmax output, sparsity, and subscale batching.

ABSTRACT

Sequential models achieve state-of-the-art results in audio, visual and textual domains with respect to both estimating the data distribution and generating high-quality samples. Efficient sampling for this class of models has however remained an elusive problem. With a focus on text-to-speech synthesis, we describe a set of general techniques for reducing sampling time while maintaining high output quality. We first describe a single-layer recurrent neural network, the WaveRNN, with a dual softmax layer that matches the quality of the state-of-the-art WaveNet model. The compact form of the network makes it possible to generate 24kHz 16-bit audio 4x faster than real time on a GPU. Second, we apply a weight pruning technique to reduce the number of weights in the WaveRNN. We find that, for a constant number of parameters, large sparse networks perform better than small dense networks and this relationship holds for sparsity levels beyond 96%. The small number of weights in a Sparse WaveRNN makes it possible to sample high-fidelity audio on a mobile CPU in real time. Finally, we propose a new generation scheme based on subscaling that folds a long sequence into a batch of shorter sequences and allows one to generate multiple samples at once. The Subscale WaveRNN produces 16 samples per step without loss of quality and offers an orthogonal method for increasing sampling efficiency.

Motivation & Objective

• Reduce sampling time for sequential neural audio models without sacrificing quality.
• Develop a compact single-layer RNN (WaveRNN) with an efficient dual softmax output.
• Explore weight pruning and structured sparsity to enable real-time or on-device synthesis.
• Introduce subscale sampling to batch generate multiple samples for increased throughput.

Proposed method

• Propose WaveRNN, a single-layer RNN with dual softmax to predict 16-bit audio samples.
• Apply weight pruning to create Sparse WaveRNN and evaluate different sparsity patterns (e.g., 4x4, 16x1 blocks).
• Implement custom GPU kernels to minimize overhead and achieve real-time sampling on GPUs.
• Introduce Subscale WaveRNN to fold long sequences into batches of shorter sequences and generate multiple samples per step.
• Experiment with Subscale Fusion to increase bits-per-step generation.
• Benchmark on 24 kHz 16-bit speech with NLL, MOS, and A/B tests.

Experimental results

Research questions

• RQ1 Can a compact RNN with a dual softmax match WaveNet quality for 16-bit audio?
• RQ2 How does structured sparsity affect synthesis quality under fixed parameter budgets?
• RQ3 Can subscale batching provide real-time or faster sampling while preserving fidelity?
• RQ4 What are the practical gains of on-device (mobile CPU) sampling for sparse WaveRNN?
• RQ5 How do different generation schemes (subscale, fused subscale) compare in throughput and quality?

Key findings

• WaveRNN (896 units) attains NLL comparable to large WaveNet and MOS similar to baseline with 4× real-time GPU synthesis (96k samples/s).
• Sparse WaveRNN with high sparsity (beyond 96%) yields better fidelity than small dense networks for the same parameter count and can run in real time on mobile CPUs.
• Subscale WaveRNN with B=16 achieves 16 samples per step with no loss in audio fidelity and enables batched sampling for throughput gains.
• On GPU, a single persistent WaveRNN kernel achieves 96,000 samples/second (WaveRNN-896) vs 8,000 samples/second for a WaveNet baseline.
• Sparse WaveRNN mobile benchmarks show real-time on-device synthesis on common mobile CPUs with 95% sparsity and 4x4 or 16x1 block structures.
• Fused Subscale WaveRNN can reach 10× real-time when generating 32 bits per step on GPUs.