[Paper Review] Multi-label Music Genre Classification from Audio, Text, and Images Using Deep Features

Sergio Oramas, Oriol Nieto+2 more|arXiv (Cornell University)|Jul 16, 2017

Music and Audio Processing25 references64 citations

TL;DR

The paper introduces MuMu, a multimodal dataset for multi-label music genre classification and analyzes deep feature-based models across audio, text, and image modalities, showing benefits from modality fusion.

ABSTRACT

Music genres allow to categorize musical items that share common characteristics. Although these categories are not mutually exclusive, most related research is traditionally focused on classifying tracks into a single class. Furthermore, these categories (e.g., Pop, Rock) tend to be too broad for certain applications. In this work we aim to expand this task by categorizing musical items into multiple and fine-grained labels, using three different data modalities: audio, text, and images. To this end we present MuMu, a new dataset of more than 31k albums classified into 250 genre classes. For every album we have collected the cover image, text reviews, and audio tracks. Additionally, we propose an approach for multi-label genre classification based on the combination of feature embeddings learned with state-of-the-art deep learning methodologies. Experiments show major differences between modalities, which not only introduce new baselines for multi-label genre classification, but also suggest that combining them yields improved results.

Motivation & Objective

• Motivate multi-label, fine-grained music genre classification beyond broad single-label schemes.
• Create MuMu, a large-scale multimodal dataset with audio, text, and image data linked to 250 genres across 31k albums.
• Develop and evaluate deep learning models for each modality (audio, text, image) and for multimodal fusion.
• Assess the benefit of label factorization and variety of evaluation metrics for multi-label genre tasks.

Proposed method

• Learn audio track representations with CNNs on log-CQTs of 15-second patches and train track-to-album aggregation followed by a shallow classifier.
• Represent album text via tf-idf Vector Space Model, with a semantic enrichment using Babelfy entity linking, feeding a feed-forward network.
• Extract cover image features using ResNet-101 pretrained on ImageNet and fine-tuned for genre prediction with a sigmoid output.
• Investigate two label embedding approaches: logistic (binary cross-entropy) and cosine (label factorization via PPMI, SVD, and low-dimensional embeddings).
• Combine modality-specific feature vectors by L2-normalizing and concatenating, then train a Multilayer Perceptron to predict multi-label outputs (logistic or cosine).
• Evaluate using AUC (per-label averaged) and Catalog Coverage@k (k=1,3,5) to measure accuracy and label diversity.

Experimental results

Research questions

• RQ1 Can deep learning-based representations from audio, text, and images surpass handcrafted features for multi-label genre classification?
• RQ2 To what extent do semantic enrichment of text and label embedding improve performance and diversity?
• RQ3 Do multimodal fusion of audio, text, and image features outperform single-modality models for MuMu genres?
• RQ4 How does dimensionality reduction of label space affect accuracy and catalog coverage?

Key findings

• Deep learning audio models substantially outperform traditional handcrafted audio features (e.g., timbre-mlp baseline).
• Text-based approaches (especially with semantic enrichment) yield strong AUC and higher catalog coverage than audio alone.
• Image-based classification lags behind other modalities but provides complementary information that improves multimodal performance.
• Multimodal fusion (A+T+I) with cosine or logistic targets generally yields the best overall AUC and coverage, outperforming single modalities.
• Label factorization (cosine) improves catalog coverage while maintaining competitive AUC, compared to logistic on several settings.
• The MuMu dataset enables large-scale multi-label, multimodal genre research with 31k albums, 147k tracks, and 447k reviews.