[Paper Review] Jukebox: A Generative Model for Music

Prafulla Dhariwal, Heewoo Jun+4 more|arXiv (Cornell University)|Apr 30, 2020

Music and Audio Processing73 references108 citations

TL;DR

Jukebox introduces a hierarchical VQ-VAE plus autoregressive Transformers to generate high-fidelity music with singing in raw audio, controllable by artist, genre, and lyrics, with coherence over minutes.

ABSTRACT

We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multi-scale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers. We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes. We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable. We are releasing thousands of non cherry-picked samples at https://jukebox.openai.com, along with model weights and code at https://github.com/openai/jukebox

Motivation & Objective

• Address long-range coherence in raw audio music generation.
• Compress raw audio with a hierarchical VQ-VAE to discrete codes.
• Model autoregressive priors over the discrete codes using Transformers.
• Enable controllable generation via conditioning signals (artist, genre, lyrics).
• Demonstrate generation across diverse genres and singing styles and release samples/code.

Proposed method

• Three-level VQ-VAE compresses 44 kHz audio into discrete codes with hop lengths 8, 32, and 128 and codebooks of size 2048.
• Train autoregressive priors (top-level and upsamplers) over the discrete codes using scalable Transformers.
• Condition priors on metadata (artist, genre, timing) and, for lyrics, use an encoder–decoder setup with attention to lyric tokens.
• Use spectral loss and random restarts to improve codebook usage and reconstruction fidelity.
• Train separate autoencoders per level to maximize information retention at each compression stage.
• Provide upsamplers that gradually reconstruct audio from higher to lower levels with conditioning on upper-level codes.

Experimental results

Research questions

• RQ1 Can a single system generate diverse, high-fidelity music with singing in raw audio across genres?
• RQ2 Does a hierarchical VQ-VAE plus autoregressive priors capture long-range musical structure over minutes?
• RQ3 How effective are conditioning signals (artist, genre, timing, lyrics) in directing generation and improving vocal alignment?
• RQ4 What is the impact of model scale and training data on singing intelligibility and musical timbre across styles?
• RQ5 How do sampling strategies (ancestral, windowed, primed) affect coherence and diversity of outputs?

Key findings

• The model can generate songs across genres like rock, hip-hop, and jazz with coherence over multiple minutes.
• Conditioning on artist, genre, and timing reduces entropy and enables style-specific generation; lyrics conditioning enables sung output.
• Increasing model capacity and 44 kHz VQ-VAE with large upsamplers improves fidelity and singing intelligibility, though singing of recognizable words remains challenging without lyrics conditioning.
• Sampling methods (ancestral, windowed, primed) allow controllable generation and continuation from existing audio segments.
• The approach yields coherent harmonies and natural prosody in many samples, with diversity and novelty observed across re-renditions, completions, and novel styles/voices.