[Paper Review] Long-form music generation with latent diffusion

Zach Evans, Julian D. Parker+4 more|arXiv (Cornell University)|Apr 16, 2024

Music Technology and Sound Studies4 citations

TL;DR

This paper presents a latent diffusion model that generates full-length, coherent music tracks up to 4 minutes and 45 seconds long by operating on a highly downsampled 21.5 Hz continuous latent representation. The model achieves state-of-the-art results in audio quality and prompt alignment without relying on semantic tokens, enabling end-to-end long-form music generation with strong structural coherence.

ABSTRACT

Audio-based generative models for music have seen great strides recently, but so far have not managed to produce full-length music tracks with coherent musical structure from text prompts. We show that by training a generative model on long temporal contexts it is possible to produce long-form music of up to 4m45s. Our model consists of a diffusion-transformer operating on a highly downsampled continuous latent representation (latent rate of 21.5Hz). It obtains state-of-the-art generations according to metrics on audio quality and prompt alignment, and subjective tests reveal that it produces full-length music with coherent structure.

Motivation & Objective

• To address the lack of long-term musical structure in existing text-conditioned music generation models, which typically generate only short segments (10–90s).
• To enable full-length music generation (up to 4m 45s) by training a generative model on extended temporal contexts.
• To eliminate dependency on semantic tokens by relying solely on latent diffusion in a highly compressed continuous latent space.
• To achieve state-of-the-art performance in audio quality and text-prompt alignment for long-form music generation.
• To demonstrate that coherent long-form music can be generated in a single forward pass using a diffusion-transformer on a compressed latent representation.

Proposed method

• The model uses a deep autoencoder to compress raw 44.1kHz stereo waveforms into a continuous latent space at 21.5 Hz, drastically reducing sequence length.
• A diffusion-transformer (DiT) operates directly in the latent space, using cross-attention for text and timing conditioning, and prepend conditioning for the diffusion timestep.
• Text conditioning is performed via a CLAP-based contrastive audio-text embedding model to align textual prompts with latent representations.
• The model is trained end-to-end on full-length music tracks (up to 285s), enabling full-context generation without autoregressive or hierarchical generation.
• The latent space is continuous and normalized, enabling high-fidelity audio reconstruction and stable diffusion sampling.
• The architecture is optimized for modern GPU VRAM, enabling 13-second inference time for 4m 45s generations.

Figure 2 : Architecture of the diffusion-transformer (DiT). Cross-attention includes timing and text conditioning. Prepend conditioning includes timing conditioning and also the signal conditioning on the current timestep of the diffusion process.

Experimental results

Research questions

• RQ1 Can a latent diffusion model generate long-form music (up to 4m 45s) with coherent musical structure without relying on semantic tokens?
• RQ2 Does training on full-length music contexts improve long-term structural coherence compared to models trained on short segments?
• RQ3 Can a highly downsampled latent representation (21.5 Hz) preserve perceptual audio quality while enabling long-context generation?
• RQ4 How does the model perform in terms of audio quality and prompt alignment compared to state-of-the-art baselines?
• RQ5 What are the risks of memorization in long-form music generation, and can they be mitigated through systematic analysis?

Key findings

• The model generates full-length music tracks of up to 4 minutes and 45 seconds with coherent musical structure, as confirmed by subjective evaluations.
• It achieves state-of-the-art performance in audio quality and text-prompt alignment metrics, outperforming prior models on long-context generation.
• The model produces high-fidelity audio with a latent rate of 21.5 Hz, demonstrating that perceptual quality is preserved even at low temporal resolution.
• Inference takes only 13 seconds on a single GPU, enabling efficient generation of long-form music in a single forward pass.
• A comprehensive memorization analysis found no evidence of model memorization, even when tested on 5,566 repeated training samples.
• The model exhibits additional capabilities such as audio-to-audio style transfer, vocal-like melody generation, and short-form sound effect synthesis.

Figure 3 : Architecture of the autoencoder.