[Paper Review] Melody Generation for Pop Music via Word Representation of Musical Properties

Andrew Shin, Léopold Crestel+7 more|arXiv (Cornell University)|Oct 31, 2017

Music and Audio Processing17 references7 citations

TL;DR

This paper proposes a novel neural melody generation model for pop music that represents each note and its properties (pitch, duration, velocity, etc.) as a unique 'word' in a sequence, enabling end-to-end generation via a recurrent architecture conditioned on chord progressions and song parts. The model achieves more human-like, structurally coherent, and perceptually pleasant melodies than prior methods, with human evaluators finding them harder to distinguish from human-written songs.

ABSTRACT

Automatic melody generation for pop music has been a long-time aspiration for both AI researchers and musicians. However, learning to generate euphonious melody has turned out to be highly challenging due to a number of factors. Representation of multivariate property of notes has been one of the primary challenges. It is also difficult to remain in the permissible spectrum of musical variety, outside of which would be perceived as a plain random play without auditory pleasantness. Observing the conventional structure of pop music poses further challenges. In this paper, we propose to represent each note and its properties as a unique `word,' thus lessening the prospect of misalignments between the properties, as well as reducing the complexity of learning. We also enforce regularization policies on the range of notes, thus encouraging the generated melody to stay close to what humans would find easy to follow. Furthermore, we generate melody conditioned on song part information, thus replicating the overall structure of a full song. Experimental results demonstrate that our model can generate auditorily pleasant songs that are more indistinguishable from human-written ones than previous models.

Motivation & Objective

• To address the challenge of generating musically coherent and pleasant pop music melodies using deep learning.
• To overcome the complexity of modeling multivariate note properties (pitch, duration, dynamics) by treating each note as a single 'word' in a sequence.
• To enforce structural integrity by conditioning generation on song part information (e.g., verse, chorus) and chord progressions.
• To reduce overfitting and improve singability by applying regularization on pitch range during training.
• To enable generation of diverse, high-quality melodies that are perceptually indistinguishable from human-composed ones.

Proposed method

• Each note is encoded as a unique 'word' combining its pitch, duration, and velocity into a single discrete token, reducing dimensionality and modeling complexity.
• A recurrent neural network (RNN) generates melodies word-by-word in a conditional autoregressive manner, conditioned on chord sequences and song part labels.
• Chord sequences and song part annotations are automatically extracted using a multinomial Hidden Markov Model (HMM) trained on a custom dataset.
• A pitch range regularization loss is introduced to penalize excessive pitch variation, promoting singability and human-like range.
• The model is trained using both standard RNN training and adversarial training with GANs to improve sample diversity.
• The model is evaluated via human listening tests and quantitative metrics, with additional ablation studies on loss functions and training schedules.

Experimental results

Research questions

• RQ1 Can modeling notes as discrete 'words' that encode multiple properties improve melody generation quality and reduce training complexity?
• RQ2 How does conditioning generation on song part information (e.g., chorus vs. verse) affect structural coherence and perceptual quality of generated melodies?
• RQ3 To what extent does pitch range regularization improve the singability and human-likeness of generated melodies?
• RQ4 Can adversarial training with GANs improve diversity in generated melodies without sacrificing audio quality?
• RQ5 How does the model compare to prior approaches in human evaluations, particularly in distinguishing generated melodies from human-composed ones?

Key findings

• The proposed word representation of musical properties significantly reduces model complexity and improves training efficiency by consolidating multivariate note features into single tokens.
• Conditioning on song part information leads to more structurally accurate melodies, with choruses exhibiting higher pitch ranges and more memorable motifs, as intended.
• The pitch range regularization loss effectively prevents the model from generating overly wide-ranging or difficult-to-sing melodies, enhancing singability.
• Human evaluation shows that melodies generated by the proposed model are significantly harder to distinguish from human-written songs than those from previous models.
• Adversarial training with GANs improved sample diversity but introduced mode collapse, resulting in repeated melodies for the same input.
• The model generalizes well to other instruments: separate models for bass, piano, and strings were successfully trained and generated in-tune, chordal tracks, though multi-track combinations remained dissonant due to lack of inter-instrument modeling.