[Paper Review] LP-MusicCaps: LLM-Based Pseudo Music Captioning

SeungHeon Doh, Keunwoo Choi+2 more|arXiv (Cornell University)|Jul 31, 2023

Music and Audio Processing4 citations

TL;DR

This paper proposes LP-MusicCaps, a large-scale pseudo-music captioning dataset generated by fine-tuning large language models (LLMs) on music tag data to produce high-quality, grammatically correct captions. The method achieves state-of-the-art performance in zero-shot and transfer learning settings, outperforming supervised baselines on BERT-Score and n-gram metrics while demonstrating improved generalization and novel caption generation.

ABSTRACT

Automatic music captioning, which generates natural language descriptions for given music tracks, holds significant potential for enhancing the understanding and organization of large volumes of musical data. Despite its importance, researchers face challenges due to the costly and time-consuming collection process of existing music-language datasets, which are limited in size. To address this data scarcity issue, we propose the use of large language models (LLMs) to artificially generate the description sentences from large-scale tag datasets. This results in approximately 2.2M captions paired with 0.5M audio clips. We term it Large Language Model based Pseudo music caption dataset, shortly, LP-MusicCaps. We conduct a systemic evaluation of the large-scale music captioning dataset with various quantitative evaluation metrics used in the field of natural language processing as well as human evaluation. In addition, we trained a transformer-based music captioning model with the dataset and evaluated it under zero-shot and transfer-learning settings. The results demonstrate that our proposed approach outperforms the supervised baseline model.

Motivation & Objective

• Address the scarcity of large-scale, publicly available music captioning datasets, which limits model training and evaluation.
• Overcome the limitations of existing datasets like MusicCaps, which are small (5,521 pairs) and not scalable for training robust models.
• Develop a cost-effective, scalable method to generate high-quality music captions using LLMs instead of expensive human annotation.
• Create a systematic evaluation framework for LLM-generated music captions to ensure semantic consistency, grammatical correctness, and lexical richness.
• Demonstrate that models pre-trained on LLM-generated captions generalize better than those trained on supervised data, even in zero-shot and transfer learning scenarios.

Proposed method

• Fine-tune a large language model (LLM) using instruction-tuning with prompts that convert music tags into natural language sentences.
• Use a prompt template: 'Describe the music with these tags: [tag1, tag2, ...]' to guide the LLM in generating coherent, descriptive captions.
• Train the LLM on a large-scale music tagging dataset (e.g., MagnaTagATune or MSD) to generate approximately 2.2 million pseudo-captions paired with 0.5 million audio clips.
• Apply a systematic filtering process to ensure semantic consistency, grammatical correctness, and lexical diversity in the generated captions.
• Train a transformer-based music captioning model using the LP-MusicCaps dataset via supervised fine-tuning and zero-shot inference.
• Use cross-attention mechanisms in the model architecture to align audio representations (mel-spectrograms) with generated text tokens, optimizing for autoregressive next-token prediction.

Figure 1 : The generation process of pseudo captions by feeding a large language model with instructions and manually-annotated labels.

Experimental results

Research questions

• RQ1 Can large language models effectively generate high-quality, natural-sounding music captions from music tags, achieving semantic and grammatical consistency?
• RQ2 Does a large-scale pseudo-music captioning dataset generated via LLMs outperform existing small, human-annotated datasets in downstream music captioning tasks?
• RQ3 To what extent do models pre-trained on LLM-generated captions generalize in zero-shot and transfer learning settings compared to supervised baselines?
• RQ4 How does the quality of LLM-generated captions compare to human-annotated captions in terms of diversity, novelty, and semantic relevance?
• RQ5 Can the use of LLMs reduce the cost and time of data collection while maintaining or improving model performance in music captioning?

Key findings

• The LP-MusicCaps dataset contains approximately 2.2 million caption-audio pairs, significantly expanding the scale of available training data for music captioning.
• Models trained on LP-MusicCaps outperform supervised baselines in BERT-Score and most n-gram metrics, especially in zero-shot and transfer learning scenarios.
• The LP-MusicCaps model achieves a notable improvement in BERT-Score over the supervised baseline, indicating better semantic understanding rather than mere word matching.
• The model demonstrates strong generalization, with a high Novel c score, indicating it generates new, unseen captions rather than memorizing training data.
• The LLM-based pseudo-captioning approach reduces reliance on expensive human-annotated data while maintaining high-quality outputs suitable for training robust music captioning models.
• Systematic evaluation confirms that LLM-generated captions are grammatically correct, semantically consistent with input tags, and rich in vocabulary.

Figure 2 : A-vs-B test results. Each method is compared to ground truth in terms of having more true positives and fewer false positives. The proposed methods (b, c, d, e) show comparable win+tie performance to ground truth.