[Paper Review] The challenge of realistic music generation: modelling raw audio at scale

Sander Dieleman, Aäron van den Oord+1 more|arXiv (Cornell University)|Jun 26, 2018

Music and Audio Processing47 references76 citations

TL;DR

This paper explores generating piano music directly in the raw audio domain by using autoregressive discrete autoencoders to capture long-range structure across tens of seconds, and compares VQ-VAE and AMAE instantiations within a hierarchical modeling setup.

ABSTRACT

Realistic music generation is a challenging task. When building generative models of music that are learnt from data, typically high-level representations such as scores or MIDI are used that abstract away the idiosyncrasies of a particular performance. But these nuances are very important for our perception of musicality and realism, so in this work we embark on modelling music in the raw audio domain. It has been shown that autoregressive models excel at generating raw audio waveforms of speech, but when applied to music, we find them biased towards capturing local signal structure at the expense of modelling long-range correlations. This is problematic because music exhibits structure at many different timescales. In this work, we explore autoregressive discrete autoencoders (ADAs) as a means to enable autoregressive models to capture long-range correlations in waveforms. We find that they allow us to unconditionally generate piano music directly in the raw audio domain, which shows stylistic consistency across tens of seconds.

Motivation & Objective

• Demonstrate that modeling music in raw audio is feasible and informative for capturing performance nuances ignored by symbolic representations.
• Address the local-structure bias of autoregressive models by introducing hierarchical ADAs to extend receptive fields.
• Propose and compare two discrete bottleneck schemes (VQ-VAE and AMAE) for stable, scalable training on music data.
• Show that multi-level autoregressive models can generate piano music with long-range musical coherence.
• Provide qualitative and quantitative analysis to assess fidelity and musicality of generated samples.

Proposed method

• Use autoregressive discrete autoencoders (ADA) to create a discrete, compressed conditioning signal with controllable information content.
• Instantiate ADAs via (i) VQ-VAE with vector quantization and (ii) AMAE with argmax-based quantization and a diversity loss.
• Attach encoder-downsampling to produce a lower-rate conditioning sequence and train a second autoregressive model on code sequences.
• Stack multiple levels (hop sizes) to enlarge the effective receptive field and enable long-range structure modeling.
• Evaluate using waveform-level NLLs, codebook perplexities, and qualitative human listening studies, with 16 kHz mu-law 8-bit quantization for audio.
• Leverage WaveNet-style architectures for encoder, modulator, and local autoregressive decoders, enabling precise control of receptive fields.

Experimental results

Research questions

• RQ1 Can autoregressive models capture long-range structure in raw audio music beyond local timbre and short-term dynamics?
• RQ2 Do hierarchical ADA architectures with larger receptive fields improve musicality and coherence in generated piano audio?
• RQ3 How do VQ-VAE and AMAE bottlenecks compare in terms of training stability, reconstruction quality, and code utilization on challenging musical data?
• RQ4 Does stacking multiple levels of ADAs yield perceptible improvements in fidelity and musicality of unconditional piano music generation?

Key findings

• ADAs with hop size 8 achieve substantially lower conditional NLLs than baseline WaveNet, enabling longer-range structure when used as conditioning.
• AMAE provides more reliable convergence than VQ-VAE on challenging musical data, though VQ-VAE can achieve better reconstruction with careful training (e.g., PBT).
• Code sequences produced by ADAs are less locally predictable than raw waveforms, allowing higher-level autoregressive modeling to capture longer-range patterns.
• Two-level and three-level hierarchies (ADAs on code sequences feeding large WaveNets) yield notable gains in perceived musicality, with multi-level models showing better harmony and structure in many samples.
• There is a trade-off: increasing long-range fidelity can come at some cost to local signal fidelity; samples with multiple ADA levels tend to be more musically coherent, though fidelity may vary across pieces.