[Paper Review] Mood Classification Using Listening Data

Filip Korzeniowski, Oriol Nieto+4 more|arXiv (Cornell University)|Oct 11, 2020

Music and Audio Processing39 references4 citations

TL;DR

This paper proposes using listening-based embeddings from user behavior data to classify music moods, demonstrating they outperform audio-based embeddings on a novel dataset of 67,000 tracks with expert mood annotations from AllMusic. The key contribution is showing that collaborative filtering signals from implicit listening data capture mood-relevant patterns more effectively than audio features, even across diverse audio models.

ABSTRACT

The mood of a song is a highly relevant feature for exploration and recommendation in large collections of music. These collections tend to require automatic methods for predicting such moods. In this work, we show that listening-based features outperform content-based ones when classifying moods: embeddings obtained through matrix factorization of listening data appear to be more informative of a track mood than embeddings based on its audio content. To demonstrate this, we compile a subset of the Million Song Dataset, totalling 67k tracks, with expert annotations of 188 different moods collected from AllMusic. Our results on this novel dataset not only expose the limitations of current audio-based models, but also aim to foster further reproducible research on this timely topic

Motivation & Objective

• To develop a large-scale, expert-annotated dataset linking music tracks to moods for reproducible research in music emotion recognition.
• To investigate whether listening-based embeddings can improve mood classification beyond audio-based models.
• To evaluate the performance of audio-based and listening-based embeddings on a shared, high-quality mood annotation dataset.
• To analyze the impact of tag frequency and model architecture on mood prediction performance.
• To explore the limitations of current audio-based models in capturing mood information not directly extractable from audio signals.

Proposed method

• Constructed the AMS dataset by matching 66,993 tracks from the Million Song Dataset's Taste Profile subset with expert mood annotations from AllMusic.
• Extracted audio-based embeddings using state-of-the-art models, including pre-trained and fine-tuned variants (e.g., MCN-MSD-A, SCC-A).
• Generated listening-based embeddings via matrix factorization of implicit listening behavior (e.g., play counts) from the same dataset.
• Trained multi-label classification models using both audio and listening embeddings as input features.
• Evaluated models using average precision per tag and analyzed performance across different tag frequencies.
• Compared correlation in tag-wise performance between audio-based and listening-based embeddings to assess shared information capture.

Experimental results

Research questions

• RQ1 Can listening-based embeddings outperform audio-based embeddings in classifying music moods on a large-scale, expert-annotated dataset?
• RQ2 How does the frequency of mood tags affect the performance of audio- and listening-based models?
• RQ3 To what extent do different audio-based models capture similar mood-related information, and how does this compare to listening-based models?
• RQ4 What role does the signal type (implicit vs. explicit feedback) play in the performance of mood classification models?
• RQ5 Are there mood categories that are inherently more difficult to predict, and do they correlate with data modality or model type?

Key findings

• Listening-based embeddings consistently outperformed all audio-based models in average precision across the mood classification task.
• The proprietary listening model (P-L) achieved higher performance than the public TP-L model, attributed to a 3,500x larger dataset and stronger explicit feedback signals.
• Audio-based models showed high correlation in tag-wise performance, indicating they learn similar, albeit suboptimal, mood representations.
• Listening-based models (TP-L and P-L) showed weaker inter-correlation, suggesting they capture distinct or more diverse mood patterns.
• Tag frequency had a moderate effect on performance, but both audio and listening models were similarly affected, with the exception of SCC-A, which suffered more from tag sparsity.
• The results suggest that current audio-based models fail to extract certain mood-relevant information present in listening behavior, implying that mood cues may not be fully encoded in audio signals alone.