[Paper Review] SongComposer: A Large Language Model for Lyric and Melody Generation in Song Composition

Shuangrui Ding, Zihan Liu+7 more|arXiv (Cornell University)|Feb 27, 2024

Music and Audio Processing4 citations

TL;DR

SongComposer is a large language model that generates coherent lyrics and melodies using symbolic song representation, enabling precise alignment between text and music. By introducing specialized tokenization for pitch, duration, and rest duration, and pretraining on a large-scale dataset (SongCompose-PT), it outperforms GPT-4 in lyric-to-melody, melody-to-lyric, and text-to-song generation tasks with superior token efficiency and human-readable output.

ABSTRACT

Creating lyrics and melodies for the vocal track in a symbolic format, known as song composition, demands expert musical knowledge of melody, an advanced understanding of lyrics, and precise alignment between them. Despite achievements in sub-tasks such as lyric generation, lyric-to-melody, and melody-to-lyric, etc, a unified model for song composition has not yet been achieved. In this paper, we introduce SongComposer, a pioneering step towards a unified song composition model that can readily create symbolic lyrics and melodies following instructions. SongComposer is a music-specialized large language model (LLM) that, for the first time, integrates the capability of simultaneously composing lyrics and melodies into LLMs by leveraging three key innovations: 1) a flexible tuple format for word-level alignment of lyrics and melodies, 2) an extended tokenizer vocabulary for song notes, with scalar initialization based on musical knowledge to capture rhythm, and 3) a multi-stage pipeline that captures musical structure, starting with motif-level melody patterns and progressing to phrase-level structure for improved coherence. Extensive experiments demonstrate that SongComposer outperforms advanced LLMs, including GPT-4, in tasks such as lyric-to-melody generation, melody-to-lyric generation, song continuation, and text-to-song creation. Moreover, we will release SongCompose, a large-scale dataset for training, containing paired lyrics and melodies in Chinese and English.

Motivation & Objective

• To develop a large language model capable of composing both lyrics and melodies in a human-like manner using symbolic music representation.
• To overcome the limitations of audio-based music modeling in LLMs by using structured, human-designed symbolic notation instead of quantized audio signals.
• To enable accurate alignment between lyrics and musical notes through a novel tuple-based representation of lyric-melody pairs.
• To pretrain the model on a large-scale, multilingual dataset (SongCompose-PT) containing lyrics, melodies, and paired data in Chinese and English.
• To enhance instruction-following capabilities through 10,000 curated QA pairs, enabling diverse song generation tasks in a unified framework.

Proposed method

• The model uses a novel tuple design to represent each lyric word with its corresponding musical attributes: pitch, duration, and rest duration, ensuring semantic clarity and alignment.
• Specialized tokens are introduced for pitch (e.g., 'F4'), duration (e.g., '1.0s'), and rest duration, enabling the LLM to interpret musical symbols directly without audio encoding.
• The model is pre-trained on SongCompose-PT, a dataset of 280K lyrics, 20K melodies, and 15K paired lyric-melody sets in Chinese and English, to build foundational music understanding.
• Instruction-following capability is fine-tuned using 10,000 QA-style dialogues that cover diverse song generation tasks, such as lyric-to-melody and text-to-song generation.
• The symbolic representation allows for efficient, flexible, and human-readable output, reducing token count and improving interpretability compared to audio-based tokenization.
• Ablation studies validate the necessity of discrete duration and note tokenization for accurate melody generation and alignment.

Figure 1: An example of lyrics and melody alignment. It involves ensuring the lyrics precisely match with one or more melody notes, annotated with precise timestamps.

Experimental results

Research questions

• RQ1 Can a large language model effectively generate coherent and musically accurate lyrics and melodies using symbolic song representation?
• RQ2 How does symbolic representation compare to audio-based tokenization in terms of efficiency, alignment, and performance in song generation?
• RQ3 What impact does a large-scale, multilingual, paired lyric-melody dataset have on the model’s ability to learn music-text relationships?
• RQ4 To what extent can instruction tuning with QA pairs improve the model’s versatility across diverse song generation tasks?
• RQ5 Does specialized tokenization for pitch, duration, and rest duration significantly enhance the model’s performance in melody and lyric generation?

Key findings

• SongComposer outperforms GPT-4 in lyric-to-melody generation, achieving a BERT Score of 0.653 and a melody duration (MD) of 2.05, compared to GPT-4’s 0.600 and 2.45.
• The inclusion of both pure-lyric and pure-melody datasets during pretraining leads to the best performance, demonstrating a synergistic effect on model learning.
• Discrete duration tokenization improves BERT Score by over 0.03 in melody-to-lyric tasks, confirming its importance for structural music understanding.
• Specialized note tokens significantly enhance performance in both lyric-to-melody and melody-to-lyric tasks, improving clarity and learning efficiency.
• The model achieves correct output formatting in 100% of cases when using full tokenization (pitch, duration, rest), whereas omission leads to failure in formatting.
• Ablation studies confirm that symbolic representation with explicit musical attributes enables more accurate and efficient song generation than implicit audio-based modeling.

Figure 2: Pipeline of paired lyric-melody data collection.