[Paper Review] Determining Song Similarity via Machine Learning Techniques and Tagging Information

Renato L. F. Cunha, Evandro Caldeira+1 more|arXiv (Cornell University)|Apr 12, 2017

Music and Audio Processing6 references3 citations

TL;DR

This paper proposes a machine learning approach to determine song similarity using only metadata (artist, title) and user-generated tags, leveraging tf-idf and k-NN models. It finds that tf-idf outperforms Word2Vec and k-NN significantly outperforms SVM and linear regression, achieving the highest R² scores on a Last.fm dataset of 5M songs.

ABSTRACT

The task of determining item similarity is a crucial one in a recommender system. This constitutes the base upon which the recommender system will work to determine which items are more likely to be enjoyed by a user, resulting in more user engagement. In this paper we tackle the problem of determining song similarity based solely on song metadata (such as the performer, and song title) and on tags contributed by users. We evaluate our approach under a series of different machine learning algorithms. We conclude that tf-idf achieves better results than Word2Vec to model the dataset to feature vectors. We also conclude that k-NN models have better performance than SVMs and Linear Regression for this problem.

Motivation & Objective

• To develop a scalable, metadata- and tag-based method for song similarity prediction without requiring audio files.
• To evaluate the effectiveness of NLP techniques like tf-idf and Word2Vec in modeling song feature vectors from textual metadata and tags.
• To compare the performance of diverse machine learning algorithms—k-NN, SVM, and linear regression—for predicting song similarity.
• To establish a ground truth similarity metric using co-occurrence of song pairs in user listening histories.
• To identify the most effective combination of feature representation and learning algorithm for song similarity prediction.

Proposed method

• The dataset was collected via the Last.fm API, including song metadata, user tags, and co-occurrence-based similarity scores for 138M song pairs.
• Text normalization was applied to remove accents, special characters, and convert Unicode to closest Latin equivalents, with multi-word terms like 'Rolling Stones' merged into single tokens.
• Feature vectors were constructed using two NLP techniques: tf-idf (treating each song’s tags as a document) and Word2Vec (using weighted averages of tag vectors with tag counts as weights).
• The k-NN model used cosine similarity between feature vectors to predict similarity, with performance evaluated using R² scores.
• The dataset was filtered into three versions: raw, s>1%, and s>2.5% to assess sensitivity to low-similarity pairs.
• Model evaluation used 100,000 sampled songs, split into training and test sets, with R² as the primary metric to assess model fit.

Experimental results

Research questions

• RQ1 Does tf-idf produce more effective song feature representations than Word2Vec when predicting similarity from metadata and tags?
• RQ2 How do k-NN, SVM, and linear regression models compare in performance for predicting song similarity using textual features?
• RQ3 Does filtering out low-similarity song pairs (s < 1% or s < 2.5%) improve model performance?
• RQ4 Is the co-occurrence-based similarity metric derived from user listening histories a valid and effective ground truth for song similarity?
• RQ5 Why does k-NN outperform more complex models like SVM and linear regression in this context?

Key findings

• tf-idf achieved better performance than Word2Vec in predicting song similarity, with higher R² scores across all data filtering conditions.
• k-NN models outperformed both SVM and linear regression, achieving the highest R² scores, particularly on the raw dataset where the best k-NN model reached an R² of 0.305 with k=10.
• The best-performing k-NN model on the raw dataset achieved an R² score of 0.305, significantly outperforming the mean predictor (R² = 0), indicating meaningful predictive power.
• SVM (SVR) performed the worst among all models, with R² scores below 0.2 in all configurations, and even negative scores in some filtered settings.
• Filtering out low-similarity pairs (s>1% or s>2.5%) degraded performance across all models, especially for k-NN, which saw R² drop to 0.105 and 0.069, respectively, suggesting that low-similarity pairs contribute to model generalization.
• The linear regression model performed poorly, with R² scores close to zero, indicating it was nearly as ineffective as predicting the mean value of the dataset.