[Paper Review] Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space

Sicheng Zhao, Yaxian Li+5 more|arXiv (Cornell University)|Aug 22, 2020

Music and Audio Processing87 references4 citations

TL;DR

This paper proposes a novel end-to-end framework, Cross-Modal Deep Continuous Metric Learning (CDCML), for emotion-based image and music matching in the continuous valence-arousal (VA) space. By jointly optimizing cross-modal similarity and single-modal VA emotion relationships in a shared embedding space, CDCML achieves state-of-the-art performance on a new large-scale dataset, IMEMNet, with 22.1% relative improvement in MSE and 5.4% in MAE over prior methods.

ABSTRACT

Both images and music can convey rich semantics and are widely used to induce specific emotions. Matching images and music with similar emotions might help to make emotion perceptions more vivid and stronger. Existing emotion-based image and music matching methods either employ limited categorical emotion states which cannot well reflect the complexity and subtlety of emotions, or train the matching model using an impractical multi-stage pipeline. In this paper, we study end-to-end matching between image and music based on emotions in the continuous valence-arousal (VA) space. First, we construct a large-scale dataset, termed Image-Music-Emotion-Matching-Net (IMEMNet), with over 140K image-music pairs. Second, we propose cross-modal deep continuous metric learning (CDCML) to learn a shared latent embedding space which preserves the cross-modal similarity relationship in the continuous matching space. Finally, we refine the embedding space by further preserving the single-modal emotion relationship in the VA spaces of both images and music. The metric learning in the embedding space and task regression in the label space are jointly optimized for both cross-modal matching and single-modal VA prediction. The extensive experiments conducted on IMEMNet demonstrate the superiority of CDCML for emotion-based image and music matching as compared to the state-of-the-art approaches.

Motivation & Objective

• To address the limitations of existing emotion-based image and music matching methods, which rely on coarse categorical emotion labels and multi-stage pipelines.
• To enable end-to-end learning of a shared latent embedding space that preserves both cross-modal similarity and single-modal emotion relationships in the continuous valence-arousal (VA) space.
• To construct a large-scale, publicly available dataset, IMEMNet, with over 140,000 image-music pairs annotated with continuous VA emotion labels.
• To jointly optimize metric learning in the embedding space and regression in the label space for both cross-modal matching and single-modal VA prediction.
• To demonstrate that preserving emotional relationships in both cross-modal and single-modal spaces improves generalization and performance.

Proposed method

• Constructs IMEMNet, a large-scale dataset of 140,000 image-music pairs with continuous valence-arousal (VA) emotion annotations.
• Proposes Cross-Modal Deep Continuous Metric Learning (CDCML), a multi-task learning framework that jointly optimizes for cross-modal similarity and single-modal VA emotion relationships.
• Uses a shared Siamese-like network architecture to embed both images and music into a shared latent space, preserving cross-modal similarity via MSE, feature-ratio, and feature-margin losses.
• Incorporates single-modal supervision by applying VA MSE loss and single-modal feature-ratio loss on the embedding space to preserve intra-modal emotional relationships.
• Employs a multi-task head to simultaneously predict matching similarity and continuous VA values from the shared embedding.
• Optimizes the entire network end-to-end using a composite loss combining cross-modal and single-modal supervisions.

Experimental results

Research questions

• RQ1 Can end-to-end learning in a shared embedding space improve emotion-based image and music matching compared to multi-stage pipelines?
• RQ2 Does preserving continuous valence-arousal (VA) emotion relationships in both cross-modal and single-modal spaces enhance matching performance?
• RQ3 To what extent does joint optimization of matching similarity and VA prediction improve generalization and robustness?
• RQ4 How does the inclusion of continuous VA supervision affect the discriminative quality of learned embeddings?
• RQ5 Can a large-scale, publicly available dataset like IMEMNet enable fair and reproducible benchmarking of emotion-based image-music matching models?

Key findings

• CDCML achieves a 22.1% relative improvement in Mean Squared Error (MSE) for matching similarity prediction compared to the best existing method, ACP-Net.
• The method shows a 5.4% relative improvement in Mean Absolute Error (MAE) for similarity prediction, demonstrating superior regression performance.
• The inclusion of cross-modal feature-ratio loss significantly improves both matching similarity prediction and single-modal VA prediction, indicating enhanced feature discriminability.
• Adding single-modal VA MSE and feature-ratio losses further boosts performance, especially on VA prediction, confirming the importance of intra-modal emotional structure.
• Qualitative visualization shows that CDCML successfully matches images and music with similar continuous VA emotions, such as pairing sad music with melancholic images.
• The ablation study confirms that joint optimization of cross-modal and single-modal supervisions leads to better generalization than using either loss alone.