[Paper Review] MuseCoco: Generating Symbolic Music from Text

Peiling Lu, Xin Xu+5 more|arXiv (Cornell University)|May 31, 2023

Music and Audio Processing4 citations

TL;DR

MuseCoco proposes a two-stage framework for generating high-quality symbolic music from text descriptions, leveraging musical attributes as a bridge between text and music. By combining self-supervised training on symbolic music data and GPT-4-generated text–attribute pairs, it achieves state-of-the-art performance in musicality, controllability, and control accuracy, with a 1.2B-parameter model outperforming baselines by 1.32 in overall score and 20% in objective control accuracy.

ABSTRACT

Generating music from text descriptions is a user-friendly mode since the text is a relatively easy interface for user engagement. While some approaches utilize texts to control music audio generation, editing musical elements in generated audio is challenging for users. In contrast, symbolic music offers ease of editing, making it more accessible for users to manipulate specific musical elements. In this paper, we propose MuseCoco, which generates symbolic music from text descriptions with musical attributes as the bridge to break down the task into text-to-attribute understanding and attribute-to-music generation stages. MuseCoCo stands for Music Composition Copilot that empowers musicians to generate music directly from given text descriptions, offering a significant improvement in efficiency compared to creating music entirely from scratch. The system has two main advantages: Firstly, it is data efficient. In the attribute-to-music generation stage, the attributes can be directly extracted from music sequences, making the model training self-supervised. In the text-to-attribute understanding stage, the text is synthesized and refined by ChatGPT based on the defined attribute templates. Secondly, the system can achieve precise control with specific attributes in text descriptions and offers multiple control options through attribute-conditioned or text-conditioned approaches. MuseCoco outperforms baseline systems in terms of musicality, controllability, and overall score by at least 1.27, 1.08, and 1.32 respectively. Besides, there is a notable enhancement of about 20% in objective control accuracy. In addition, we have developed a robust large-scale model with 1.2 billion parameters, showcasing exceptional controllability and musicality.

Motivation & Objective

• To enable efficient, controllable, and user-friendly symbolic music generation from natural language text descriptions.
• To overcome limitations in existing text-to-music systems, such as poor controllability, lack of editability, and reliance on large paired text-music datasets.
• To improve model generalization and performance by leveraging self-supervised learning on symbolic music data and synthetic text–attribute pairs.
• To support diverse users—both musically trained and non-experts—through dual control modes: text-to-attribute or direct attribute input.
• To establish a scalable, data-efficient system that maintains high musical quality and precise control across multiple musical attributes.

Proposed method

• The system uses a two-stage pipeline: first, text-to-attribute understanding, where GPT-4 generates coherent text descriptions from predefined attribute templates.
• Second, attribute-to-music generation, where a large transformer-based model generates symbolic music conditioned on the extracted musical attributes.
• Musical attributes (e.g., key, time signature, tempo, instrument, style) are directly extracted from existing symbolic music sequences, enabling self-supervised pretraining.
• Synthetic paired data is created by combining attribute templates into natural language prompts using GPT-4, reducing dependency on human-annotated text-music pairs.
• The model supports both text-conditioned and attribute-conditioned generation, enabling flexible user interaction.
• A 1.2-billion-parameter xlarge model is trained with the same setup as the large model, enabling high-fidelity and controllable music generation.

Experimental results

Research questions

• RQ1 Can a two-stage framework that decouples text understanding from music generation improve controllability and musical quality in text-to-symbolic music generation?
• RQ2 To what extent can self-supervised learning on symbolic music data reduce reliance on expensive paired text-music datasets?
• RQ3 How effective is GPT-4 in synthesizing high-quality, coherent text descriptions from structured musical attributes for training?
• RQ4 Does increasing model size significantly improve control accuracy and musicality in text-to-music generation?
• RQ5 Can the system support both non-expert users (via text input) and expert users (via direct attribute input), enhancing inclusivity and usability?

Key findings

• MuseCoco outperforms baseline systems by at least 1.27 in musicality, 1.08 in controllability, and 1.32 in overall score.
• The system achieves a 20% improvement in objective control accuracy compared to baselines, demonstrating strong precision in attribute adherence.
• The 1.2-billion-parameter xlarge model achieves 87.15% average objective attribute control accuracy, a significant improvement over the large model’s 83.63%.
• Professional musicians reported that MuseCoco reduces composition time by up to two weeks and provides strong creative inspiration, particularly in arrangement and genre blending.
• Musicians noted that generated music closely resembles human compositions, with coherent motifs, harmonic structure, and expressive phrasing, especially in complex styles like Bach or genre hybrids.
• The system enables effective editing and customization, with music teachers finding it highly useful for generating pedagogical examples.