[Paper Review] Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion

Yujia Huang, Adishree Ghatare+7 more|arXiv (Cornell University)|Feb 22, 2024

Music and Audio Processing4 citations

TL;DR

This paper introduces Stochastic Control Guidance (SCG), a plug-and-play method for non-differentiable rule-guided symbolic music generation using diffusion models. SCG enables training-free, high-resolution (10ms) music generation with superior controllability and quality by sampling multiple trajectory realizations at each step and selecting the rule-compliant one, outperforming state-of-the-art baselines in rule adherence and musical quality.

ABSTRACT

We study the problem of symbolic music generation (e.g., generating piano rolls), with a technical focus on non-differentiable rule guidance. Musical rules are often expressed in symbolic form on note characteristics, such as note density or chord progression, many of which are non-differentiable which pose a challenge when using them for guided diffusion. We propose Stochastic Control Guidance (SCG), a novel guidance method that only requires forward evaluation of rule functions that can work with pre-trained diffusion models in a plug-and-play way, thus achieving training-free guidance for non-differentiable rules for the first time. Additionally, we introduce a latent diffusion architecture for symbolic music generation with high time resolution, which can be composed with SCG in a plug-and-play fashion. Compared to standard strong baselines in symbolic music generation, this framework demonstrates marked advancements in music quality and rule-based controllability, outperforming current state-of-the-art generators in a variety of settings. For detailed demonstrations, code and model checkpoints, please visit our project website: https://scg-rule-guided-music.github.io/.

Motivation & Objective

• To address the challenge of guiding pre-trained diffusion models with non-differentiable musical rules such as chord progressions and note density.
• To enable training-free, plug-and-play rule guidance that does not require backpropagation or model fine-tuning.
• To develop a high-resolution (10ms) latent diffusion architecture for expressive piano roll generation with velocity and sustain pedal.
• To achieve superior music quality and rule-based controllability compared to existing strong baselines.
• To provide a flexible, interpretable, and musician-friendly tool for symbolic music generation.

Proposed method

• Proposes Stochastic Control Guidance (SCG), a novel algorithm inspired by path integral control theory for optimal control in stochastic dynamical systems.
• At each sampling step, SCG generates multiple realizations of the next step and selects the one yielding the cleanest sample that best satisfies the rule function.
• SCG only requires forward evaluation of rule functions, making it compatible with non-differentiable, black-box, or symbolic rule APIs.
• Employs a latent diffusion model with a transformer backbone to generate high-resolution (10ms) symbolic music including velocity and pedal information.
• The framework is fully plug-and-play: SCG can be applied to any pre-trained diffusion model without retraining.
• Uses a latent space representation that enables efficient, high-fidelity music generation at fine temporal resolution.

Experimental results

Research questions

• RQ1 Can non-differentiable musical rules such as chord progressions and note density be effectively used for guidance in diffusion models?
• RQ2 Can a training-free, plug-and-play method achieve better rule adherence and music quality than existing fine-tuned or conditional baselines?
• RQ3 Can high-resolution (10ms) symbolic music generation be achieved with a latent diffusion model while maintaining expressive dynamics and pedal control?
• RQ4 How does the proposed SCG method compare to gradient-based guidance in terms of controllability and sample quality when rules are non-differentiable?
• RQ5 To what extent does SCG enable interpretable and flexible control for human composers in music generation?

Key findings

• SCG achieves state-of-the-art performance in symbolic music generation, outperforming strong baselines in both music quality and rule-based controllability.
• The method enables effective, training-free guidance with non-differentiable rules, a capability previously unattainable with gradient-based approaches.
• The latent diffusion model with 10ms resolution successfully generates expressive piano rolls with velocity and sustain pedal information.
• Human evaluation shows the generated music is highly rated for coherence, harmony, texture, and overall appeal, with a significant majority rating it as 'Good' or 'Very Good'.
• The framework demonstrates marked improvements in chord progression alignment, with a high proportion of samples rated as 'Mostly Aligned' or 'Perfectly Aligned'.
• The method enables flexible, interpretable, and controllable music generation, making it suitable as a compositional tool for musicians.