[Paper Review] A Novel Audio Representation for Music Genre Identification in MIR

Navin Kamuni, Mayank Jindal+3 more|arXiv (Cornell University)|Apr 1, 2024

Music and Audio Processing4 citations

TL;DR

This paper evaluates a novel audio representation derived from the Jukebox generative model for music genre identification in Music Information Retrieval (MIR). Using a transformer-based classifier on a dataset comparable to SOTA, the study finds that Jukebox's representation does not outperform standard Mel spectrograms, likely due to its lack of alignment with human auditory perception, despite its deep vector quantization approach.

ABSTRACT

For Music Information Retrieval downstream tasks, the most common audio representation is time-frequency-based, such as Mel spectrograms. In order to identify musical genres, this study explores the possibilities of a new form of audio representation one of the most usual MIR downstream tasks. Therefore, to discretely encoding music using deep vector quantization; a novel audio representation was created for the innovative generative music model i.e. Jukebox. The effectiveness of Jukebox's audio representation is compared to Mel spectrograms using a dataset that is almost equivalent to State-of-the-Art (SOTA) and an almost same transformer design. The results of this study imply that, at least when the transformers are pretrained using a very modest dataset of 20k tracks, Jukebox's audio representation is not superior to Mel spectrograms. This could be explained by the fact that Jukebox's audio representation does not sufficiently take into account the peculiarities of human hearing perception. On the other hand, Mel spectrograms are specifically created with the human auditory sense in mind.

Motivation & Objective

• To investigate whether Jukebox's deep vector-quantized audio representation improves music genre identification in MIR.
• To compare the performance of Jukebox's representation against standard Mel spectrograms under identical model and data conditions.
• To assess whether the generative pretraining of Jukebox yields a more effective representation for downstream MIR tasks.
• To explore whether the lack of human auditory perception modeling in Jukebox's representation limits its effectiveness.
• To determine if the representation's performance is sensitive to the scale of pretraining data.

Proposed method

• The study employs a transformer-based classifier architecture identical for both representations to ensure fair comparison.
• Jukebox's audio representation is extracted via its learned vector-quantized VAE, producing discrete latent codes from raw audio.
• Mel spectrograms are computed using standard Mel-scaled filterbanks, a well-established baseline in MIR.
• The model is fine-tuned on a dataset of approximately 20,000 tracks, a modest pretraining size.
• Performance is evaluated using standard metrics for music genre classification, such as accuracy and F1-score.
• The comparison is conducted under identical training and evaluation protocols to isolate the effect of the audio representation.

Experimental results

Research questions

• RQ1 Does Jukebox’s vector-quantized audio representation outperform Mel spectrograms in music genre classification?
• RQ2 How does the performance of Jukebox’s representation compare to Mel spectrograms when using the same transformer architecture and dataset size?
• RQ3 To what extent does the lack of human auditory perception modeling in Jukebox’s representation affect its performance in MIR tasks?
• RQ4 Is the performance gap between representations sensitive to the scale of the pretraining data?
• RQ5 Can a generative model’s learned representation be as effective as hand-crafted representations like Mel spectrograms for downstream MIR tasks?

Key findings

• Jukebox’s audio representation did not achieve superior performance compared to Mel spectrograms in music genre identification.
• The performance of the transformer model using Jukebox’s representation was consistently lower than with Mel spectrograms across evaluation metrics.
• The study attributes the underperformance to Jukebox’s representation not being optimized for human auditory perception, unlike Mel spectrograms.
• Even with a modest pretraining dataset of 20,000 tracks, Mel spectrograms maintained a performance advantage.
• The results suggest that hand-crafted representations like Mel spectrograms remain competitive or superior in this downstream task.
• The findings indicate that generative pretraining alone does not guarantee better representation learning for MIR tasks.