[Paper Review] Learning Audio Embeddings with User Listening Data for Content-based Music Recommendation

Ke Chen, Beici Liang+2 more|arXiv (Cornell University)|Oct 29, 2020

Music and Audio Processing19 references4 citations

TL;DR

This paper proposes a dual-branch model that learns audio embeddings (AEs) by combining user listening history and demographics with metric learning via Siamese networks. Using user embeddings as anchors, the model trains on liked/disliked track pairs to generate content-based AEs, achieving state-of-the-art performance in music recommendation (precision: 0.773, AUC: 0.849) and improved genre classification accuracy.

ABSTRACT

Personalized recommendation on new track releases has always been a challenging problem in the music industry. To combat this problem, we first explore user listening history and demographics to construct a user embedding representing the user's music preference. With the user embedding and audio data from user's liked and disliked tracks, an audio embedding can be obtained for each track using metric learning with Siamese networks. For a new track, we can decide the best group of users to recommend by computing the similarity between the track's audio embedding and different user embeddings, respectively. The proposed system yields state-of-the-art performance on content-based music recommendation tested with millions of users and tracks. Also, we extract audio embeddings as features for music genre classification tasks. The results show the generalization ability of our audio embeddings.

Motivation & Objective

• Address the cold-start problem in music recommendation for new track releases by leveraging audio and user data.
• Improve user preference modeling by incorporating both listening history and demographic features into user embeddings.
• Develop a metric learning framework that uses user embeddings as anchors to learn discriminative audio embeddings from audio content.
• Evaluate the generalization of learned audio embeddings across music recommendation and genre classification tasks.
• Demonstrate that user-anchored metric learning yields more effective audio representations than audio-only baselines.

Proposed method

• Train a user branch using a YouTubeDNN-inspired architecture to generate 40-dimensional user embeddings (UE) from listening history and demographics.
• Use explicit and implicit feedback (e.g., playlist additions) as supervision in the user branch, with cross-entropy loss for classification.
• Implement a Siamese network in the audio branch that shares weights across two towers to compare audio embeddings of liked and disliked tracks.
• Feed log-mel spectrograms to the Siamese CNNs, with ReLU activations after each convolutional layer and a 40-dimensional output as the audio embedding.
• Apply metric learning with triplet loss (contrastive loss) to push liked tracks' embeddings closer to the user embedding and push disliked ones farther.
• Use negative sampling during training to efficiently optimize the Siamese network on large-scale user-track interaction data.

Experimental results

Research questions

• RQ1 Can user embeddings derived from comprehensive listening history and demographics improve the quality of audio embeddings for content-based recommendation?
• RQ2 Does using user embeddings as anchors in metric learning enhance the discriminative power of audio embeddings compared to audio-only learning?
• RQ3 How well do the learned audio embeddings generalize to downstream tasks like music genre classification?
• RQ4 What is the impact of context window duration (e.g., 10s) on the performance of the audio embedding model?
• RQ5 Can the proposed model outperform existing state-of-the-art methods in both music recommendation and genre classification?

Key findings

• The proposed model achieved a precision of 0.773 and AUC of 0.849 on a large-scale dataset with 60 million user-track interactions, outperforming baseline models.
• In genre classification, the model improved SVM accuracy from 0.7653 (without AE) to 0.8013 on the GTZAN dataset, and from 0.5918 to 0.6148 on FMA-small.
• On the MIREX 2020 genre classification task, the model achieved an average accuracy of 0.7474, with top performance in classifying country, rap/hip-hop, and K-pop ballad genres.
• Extending the context window to 10 seconds improved both precision and AUC, indicating that longer audio context enhances embedding quality.
• The audio embeddings learned via user-anchored metric learning generalized well to both recommendation and classification tasks, demonstrating strong transferability.
• The model's performance was robust across diverse music genres, with particularly strong results in K-pop and rap/hip-hop, suggesting sensitivity to genre-specific audio patterns.