[Paper Review] ChatMusician: Understanding and Generating Music Intrinsically with LLM

Ruibin Yuan, Hanfeng Lin+33 more|arXiv (Cornell University)|Feb 25, 2024

Machine Learning in Materials Science4 citations

TL;DR

ChatMusician is an open-source large language model fine-tuned on ABC notation to enable intrinsic music understanding and generation using only a text tokenizer, without external multimodal components. It outperforms GPT-4 and GPT-3.5 in music generation and reasoning tasks, achieves a 99.6% ABC notation parsing success rate, and maintains or slightly improves general language abilities, as shown by a 46.80 MMLU score.

ABSTRACT

While Large Language Models (LLMs) demonstrate impressive capabilities in text generation, we find that their ability has yet to be generalized to music, humanity's creative language. We introduce ChatMusician, an open-source LLM that integrates intrinsic musical abilities. It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language. ChatMusician can understand and generate music with a pure text tokenizer without any external multi-modal neural structures or tokenizers. Interestingly, endowing musical abilities does not harm language abilities, even achieving a slightly higher MMLU score. Our model is capable of composing well-structured, full-length music, conditioned on texts, chords, melodies, motifs, musical forms, etc, surpassing GPT-4 baseline. On our meticulously curated college-level music understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs can be an excellent compressor for music, but there remains significant territory to be conquered. We release our 4B token music-language corpora MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.

Motivation & Objective

• To develop a large language model with intrinsic musical capabilities using only text-based tokenization.
• To unify music generation and understanding within a single LLM without relying on external multimodal architectures.
• To evaluate the model’s performance on symbolic music reasoning and composition tasks at a college-level standard.
• To release a comprehensive music-language dataset, benchmark, and model to advance research in music and LLMs.

Proposed method

• Continual pre-training and fine-tuning of LLaMA2 on a 4B-token music-language corpus using ABC notation as the text-compatible music representation.
• Treating music as a second language by encoding musical structures into token sequences compatible with standard LLM tokenizers.
• Designing and applying handcrafted instruction templates for diverse music generation tasks, including chord, motif, and form conditioning.
• Introducing MusicTheoryBench, a college-level symbolic music understanding benchmark for evaluating music reasoning and theory comprehension.
• Employing zero-shot evaluation on music generation and understanding tasks to assess generalization and coherence.
• Analyzing memorization and hallucination effects via comparison of generated ABC notation with ground truth training sequences.

Experimental results

Research questions

• RQ1 Can an LLM achieve strong music generation and understanding capabilities when trained solely on text-compatible music representations like ABC notation?
• RQ2 Does integrating musical capabilities into an LLM degrade or enhance its general language abilities?
• RQ3 How does the model perform on complex, symbolic music reasoning tasks requiring knowledge of harmony, form, and structure?
• RQ4 To what extent does the model memorize training data, and what are the implications for copyright and educational use?

Key findings

• ChatMusician achieved a 99.6% success rate in parsing generated ABC notation, significantly outperforming GPT-4 (94.6%) and GPT-3.5 (65.4%).
• The model surpassed GPT-4 and established baselines in all five music generation tasks, with higher average percentile scores across all conditions.
• ChatMusician achieved a 46.80 MMLU score, slightly higher than LLaMA2-7B-Base (46.79), indicating no degradation and possible enhancement of general language abilities.
• On the MusicTheoryBench benchmark, ChatMusician outperformed LLaMA2 and GPT-3.5 in zero-shot settings, revealing its superior symbolic music reasoning capabilities.
• The model exhibited minimal memorization, with only 0.02% of training sequences fully replicated and 0.24% partially matched (≥80% token overlap).
• Despite strong performance, the model shows a bias toward Irish music due to dataset composition and struggles with open-ended generation due to limited instruction diversity.