[Paper Review] A Hybrid of Deep Audio Feature and i-vector for Artist Recognition

Jiyoung Park, Donghyun Kim+3 more|arXiv (Cornell University)|Jul 24, 2018

Music and Audio Processing7 references4 citations

TL;DR

This paper proposes a late fusion hybrid model combining deep convolutional neural network (DCNN)-extracted audio features and i-vector representations for music artist recognition. By leveraging the discriminative power of DCNN and the generative robustness of i-vector, the method achieves state-of-the-art performance, particularly excelling in singer recognition and outperforming individual models through complementary feature learning.

ABSTRACT

Artist recognition is a task of modeling the artist's musical style. This problem is challenging because there is no clear standard. We propose a hybrid method of the generative model i-vector and the discriminative model deep convolutional neural network. We show that this approach achieves state-of-the-art performance by complementing each other. In addition, we briefly explain the advantages and disadvantages of each approach.

Motivation & Objective

• To improve artist recognition in music information retrieval by combining deep learning and traditional speaker modeling techniques.
• To investigate the complementary strengths of discriminative deep audio features and generative i-vector representations in modeling musical artist style.
• To evaluate the effectiveness of early versus late fusion strategies in combining DCNN and i-vector features for artist and singer recognition.
• To analyze how model performance varies with the number of training artists and the nature of audio content (e.g., vocal focus).
• To explore the potential of the hybrid model for future applications such as recommending similar artists.

Proposed method

• Extract deep audio features using a 5-convolutional-layer DCNN trained on 3-second mel-spectrograms with 128 bins, using the final 256-dimensional hidden layer as the feature representation.
• Generate i-vectors using 20-dimensional Mel-Frequency Cepstral Coefficients (MFCCs) and a 256-component Gaussian Mixture Model (GMM), followed by Probabilistic Linear Discriminant Analysis (PLDA) for scoring.
• Implement early fusion by concatenating the 256-dim DCNN feature and 256-dim i-vector into a single 512-dim vector before scoring.
• Implement late fusion by averaging the PLDA scores from both DCNN and i-vector models, allowing each to contribute independently to the final decision.
• Train and evaluate models on the Million Song Dataset, filtering 20 songs per artist and using 500 unseen artists for evaluation.
• Use average pooling of 15 enrollment tracks to form a single artist model, and evaluate using Equal Error Rate (EER) for verification and classification accuracy for identification.

Experimental results

Research questions

• RQ1 How do the performance characteristics of i-vector and DCNN differ as the number of training artists increases?
• RQ2 In what ways do i-vector and DCNN complement each other in modeling musical artist identity?
• RQ3 Does early fusion or late fusion of DCNN and i-vector yield better performance in artist and singer recognition tasks?
• RQ4 How does the focus on vocal content (e.g., singer recognition) affect the relative performance of i-vector and DCNN?
• RQ5 Can the hybrid model achieve state-of-the-art performance in artist recognition by leveraging the strengths of both discriminative and generative models?

Key findings

• Late fusion of DCNN and i-vector achieved the best performance across all tasks, with a 3.241% EER and 76.0% accuracy in singer recognition, significantly outperforming individual models.
• DCNN performance improved steadily with more training artists, while i-vector performance plateaued early, indicating i-vector's insensitivity to training data size due to its unsupervised nature.
• i-vector showed superior performance in singer recognition (8.257% EER) compared to artist recognition (10.785% EER), suggesting its strength in modeling vocal characteristics.
• Early fusion degraded performance compared to both standalone models, likely due to feature confusion from combining distinct representations before scoring.
• Confusion matrices revealed that DCNN struggled to form stable models for some artists with fewer training examples, while i-vector formed models early but suffered from increased inter-artist similarity as training size grew.
• The hybrid model demonstrated strong complementarity: DCNN captured higher-level musical features, while i-vector excelled in voice-based distinctions, especially in vocal-focused tasks.