[Paper Review] Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

Jesse Engel, Cinjon Resnick+5 more|arXiv (Cornell University)|Apr 5, 2017

Music and Audio Processing26 references298 citations

TL;DR

The paper introduces a WaveNet-style autoencoder to learn temporal embeddings for audio synthesis and evaluates it on NSynth, showing superior reconstruction and meaningful timbre interpolation compared to a spectral autoencoder baseline.

ABSTRACT

Generative models in vision have seen rapid progress due to algorithmic improvements and the availability of high-quality image datasets. In this paper, we offer contributions in both these areas to enable similar progress in audio modeling. First, we detail a powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform. Second, we introduce NSynth, a large-scale and high-quality dataset of musical notes that is an order of magnitude larger than comparable public datasets. Using NSynth, we demonstrate improved qualitative and quantitative performance of the WaveNet autoencoder over a well-tuned spectral autoencoder baseline. Finally, we show that the model learns a manifold of embeddings that allows for morphing between instruments, meaningfully interpolating in timbre to create new types of sounds that are realistic and expressive.

Motivation & Objective

• Develop a WaveNet-style autoencoder that captures long-range temporal structure without external conditioning.
• Create a large-scale, high-quality NSynth dataset of musical notes for training and evaluation.
• Demonstrate improved reconstruction quality and perceptual fidelity over a spectral autoencoder baseline.
• Show that the learned embeddings support meaningful interpolation across timbre and dynamics.
• Explore how embeddings encode pitch and timbre and how conditioning affects representations.

Proposed method

• Design a temporal encoder: a 30-layer residual dilated convolutional network producing time-distributed embeddings Z.
• Design a WaveNet decoder that is conditioned on the embedded Z by biasing each layer with projections of Z and upsampling temporally to audio rate.
• Train with mu-law 8-bit quantization and autoregressive generation conditioned on Z for each output sample.
• Compare against a convolutional spectral autoencoder baseline trained on spectrogram representations.
• Use NSynth as the dataset: ~306k notes, 4 seconds each, 16 kHz, across ~1000 instruments.

Experimental results

Research questions

• RQ1 Can a WaveNet-style autoencoder learn long-term temporal structure without external conditioning for audio synthesis?
• RQ2 Do learned embeddings capture timbre and dynamics such that interpolation between instruments yields realistic sounds?
• RQ3 How does the WaveNet autoencoder compare to a spectral autoencoder baseline in terms of reconstruction quality and perceptual fidelity?
• RQ4 To what extent does pitch conditioning disentangle pitch from timbre in the learned embedding Z?
• RQ5 What are the properties and generalization capabilities of the temporal embeddings beyond the training context?

Key findings

• WaveNet autoencoder achieves higher perceptual and qualitative fidelity than the spectral autoencoder baseline in reconstruction tasks.
• A classifier trained on embeddings shows higher pitch and quality prediction accuracy for WaveNet reconstructions than for baseline reconstructions, indicating better preservation of timbre and pitch cues.
• Embeddings support meaningful timbre/dynamics interpolation between instruments, producing perceptually coherent blends rather than simple superpositions.
• Pitch conditioning during training tends to disentangle pitch from timbre to varying degrees, with larger embedding sizes showing less decoupling.
• Temporal embeddings exhibit a driving-function-like behavior, enabling generalization to note sequences longer than those seen during training.