[Paper Review] Emotion Embedding Spaces for Matching Music to Stories

Minz Won, Justin Salamon+3 more|arXiv (Cornell University)|Nov 26, 2021

Music and Audio Processing50 references4 citations

TL;DR

This paper proposes a cross-modal text-to-music retrieval framework that bridges heterogeneous emotion vocabularies using emotion embedding spaces, combining manually defined (valence-arousal) and data-driven approaches (Word2Vec, metric learning). It demonstrates that data-driven metric learning—particularly a three-branch model—achieves performance comparable to valence-arousal baselines while preserving emotional neighborhood structure and enabling generalization to diverse emotion vocabularies.

ABSTRACT

Content creators often use music to enhance their stories, as it can be a powerful tool to convey emotion. In this paper, our goal is to help creators find music to match the emotion of their story. We focus on text-based stories that can be auralized (e.g., books), use multiple sentences as input queries, and automatically retrieve matching music. We formalize this task as a cross-modal text-to-music retrieval problem. Both the music and text domains have existing datasets with emotion labels, but mismatched emotion vocabularies prevent us from using mood or emotion annotations directly for matching. To address this challenge, we propose and investigate several emotion embedding spaces, both manually defined (e.g., valence/arousal) and data-driven (e.g., Word2Vec and metric learning) to bridge this gap. Our experiments show that by leveraging these embedding spaces, we are able to successfully bridge the gap between modalities to facilitate cross modal retrieval. We show that our method can leverage the well established valence-arousal space, but that it can also achieve our goal via data-driven embedding spaces. By leveraging data-driven embeddings, our approach has the potential of being generalized to other retrieval tasks that require broader or completely different vocabularies.

Motivation & Objective

• To address the challenge of matching music to text based on shared emotional content despite mismatched emotion vocabularies across modalities.
• To formalize emotion-based text-to-music retrieval as a cross-modal retrieval problem with long-form text inputs.
• To evaluate and compare multiple emotion embedding space strategies—both handcrafted and data-driven—for effective cross-modal alignment.
• To develop a method that generalizes beyond predefined emotion spaces, enabling application to diverse or novel emotion vocabularies.

Proposed method

• Proposes six deep learning strategies: classification, multi-head classification, valence-arousal regression, Word2Vec regression, two-branch metric learning, and three-branch metric learning.
• Uses pre-trained text encoders (e.g., BERT-like models) and audio encoders (e.g., CNNs on mel-spectrograms) to embed text and music into shared emotion spaces.
• Employs UMAP for 2D visualization of high-dimensional embedding spaces to analyze structural properties like neighborhood preservation.
• Applies metric learning with three branches to jointly embed text, music, and mood tags, learning a shared space that preserves intra-modality emotional proximity.
• Uses precision at five (P@5) and mean reciprocal rank (mRR) as evaluation metrics on multiple datasets including Alm’s text/mood and AudioSet music/mood.
• Leverages transfer learning and focal loss in audio emotion tagging to improve robustness and performance on imbalanced emotion distributions.

Experimental results

Research questions

• RQ1 Can a shared emotion embedding space be effectively learned to enable cross-modal text-to-music retrieval despite mismatched emotion vocabularies?
• RQ2 How do data-driven embedding spaces (e.g., Word2Vec, metric learning) compare to manually defined spaces (e.g., valence-arousal) in terms of retrieval performance and generalization?
• RQ3 To what extent does the three-branch metric learning model preserve the neighborhood structure of emotions within each modality compared to other methods?
• RQ4 Can the proposed method generalize to retrieval tasks with broader or completely different emotion vocabularies than those used in training?

Key findings

• The valence-arousal space serves as a strong baseline for cross-modal retrieval, successfully aligning text and music based on emotional dimensions.
• The three-branch metric learning model achieves retrieval performance comparable to the valence-arousal baseline while learning a data-driven, generalizable embedding space.
• The three-branch model preserves intra-modality neighborhood structure—emotions with similar meanings cluster together in the embedding space—unlike one-way or two-branch models.
• Word2Vec regression produces discriminative embeddings but lacks continuity, while two-branch metric learning fails to preserve structural relationships due to lack of modality-specific supervision.
• The proposed method generalizes beyond predefined emotion spaces, enabling application to new or diverse emotion vocabularies without requiring manual mapping.
• Qualitative analysis confirms that emotionally similar moods (e.g., 'scary' and 'fearful') are spatially close in the learned embedding space, especially in the three-branch metric learning model.