[Paper Review] Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm

Ke Chen, Cheng-i Wang+2 more|arXiv (Cornell University)|Aug 4, 2020

Music and Audio Processing23 references22 citations

TL;DR

Music SketchNet introduces a controllable music generation framework that factorizes pitch and rhythm into disentangled latent representations using a variational autoencoder (SketchVAE), enabling users to guide music completion via partial pitch or rhythm sketches. The model outperforms state-of-the-art methods in both objective metrics and subjective listening tests, achieving high accuracy in following user-specified contours while maintaining musical coherence.

ABSTRACT

Drawing an analogy with automatic image completion systems, we propose Music SketchNet, a neural network framework that allows users to specify partial musical ideas guiding automatic music generation. We focus on generating the missing measures in incomplete monophonic musical pieces, conditioned on surrounding context, and optionally guided by user-specified pitch and rhythm snippets. First, we introduce SketchVAE, a novel variational autoencoder that explicitly factorizes rhythm and pitch contour to form the basis of our proposed model. Then we introduce two discriminative architectures, SketchInpainter and SketchConnector, that in conjunction perform the guided music completion, filling in representations for the missing measures conditioned on surrounding context and user-specified snippets. We evaluate SketchNet on a standard dataset of Irish folk music and compare with models from recent works. When used for music completion, our approach outperforms the state-of-the-art both in terms of objective metrics and subjective listening tests. Finally, we demonstrate that our model can successfully incorporate user-specified snippets during the generation process.

Motivation & Objective

• To enable intuitive, user-guided music generation by allowing partial sketches of pitch or rhythm to condition music completion.
• To address the limitation of existing models that require full musical tracks for conditioning, by introducing a sketch-based interface for music generation.
• To decouple latent representations of pitch and rhythm in music to support fine-grained, controllable generation.
• To improve music inpainting by incorporating user preferences without requiring complete musical inputs.
• To evaluate the effectiveness of user control in generating coherent, musically plausible music through both quantitative and subjective assessments.

Proposed method

• SketchVAE is a variational autoencoder that factorizes music measures into disentangled latent representations for pitch contour and rhythm using a factorized inference network.
• SketchInpainter uses stacked recurrent networks to predict latent representations for missing measures based on past and future context.
• SketchConnector integrates user-specified pitch or rhythm sketches with the latent predictions from SketchInpainter to guide the final generation.
• The joint probability model combines the decoder (SketchVAE), the inpainter (SketchInpainter), and the connector (SketchConnector) into a unified framework for conditional generation.
• The model is trained end-to-end on Irish folk music data, with latent space modeling enabling disentangled control over pitch and rhythm.
• A virtual control experiment evaluates user influence by measuring pitch and rhythm accuracy between generated outputs and user-specified sketches.

Experimental results

Research questions

• RQ1 Can factorized latent representations of pitch and rhythm enable more intuitive and controllable music generation compared to end-to-end models?
• RQ2 How effectively can users guide music generation using partial sketches of pitch or rhythm without providing complete musical tracks?
• RQ3 Does the inclusion of user-specified pitch or rhythm sketches lead to statistically significant improvements in generation quality compared to baseline inpainting models?
• RQ4 To what extent does the model preserve musical coherence and structural consistency when guided by user sketches?
• RQ5 Can the model maintain high accuracy in following user-specified pitch and rhythm patterns while generating musically plausible content?

Key findings

• Music SketchNet outperforms the state-of-the-art Music InpaintNet in both objective metrics and subjective listening tests, particularly in structure and overall musicality.
• Subjective evaluations showed that SketchNet scored significantly higher than Music InpaintNet in structure (p < 0.05) and overall musicality (p < 0.05), with no significant difference in complexity (p = 0.364).
• In the virtual control experiment, the model achieved 97.3% accuracy in following user-specified rhythm sketches and 88.1% accuracy in following pitch contours.
• The model successfully generated coherent music that matched user-specified pitch and rhythm sketches, even when the original melody had scale shifts.
• The disentangled latent space enabled meaningful control: users could independently influence pitch or rhythm, resulting in diverse, contextually appropriate outputs.
• The framework demonstrated practical utility in interactive music sketching, where users could guide generation with minimal input while maintaining musical quality.