[Paper Review] Towards Deep Modeling of Music Semantics using EEG Regularizers

Francisco Raposo, David Martins de Matos+3 more|arXiv (Cornell University)|Dec 14, 2017

Music and Audio Processing24 references3 citations

TL;DR

This paper proposes a novel end-to-end deep neural network that jointly models music audio and EEG signals using Deep Canonical Correlation Analysis (DCCA) to learn a shared semantic embedding space. By leveraging brainwave data as a perceptual regularizer, the model achieves state-of-the-art performance in audio-lyrics cross-modal retrieval—outperforming Spotify features and matching a SOTA model trained on 700× more data—using only 2.5 hours of audio-EEG data.

ABSTRACT

Modeling of music audio semantics has been previously tackled through learning of mappings from audio data to high-level tags or latent unsupervised spaces. The resulting semantic spaces are theoretically limited, either because the chosen high-level tags do not cover all of music semantics or because audio data itself is not enough to determine music semantics. In this paper, we propose a generic framework for semantics modeling that focuses on the perception of the listener, through EEG data, in addition to audio data. We implement this framework using a novel end-to-end 2-view Neural Network (NN) architecture and a Deep Canonical Correlation Analysis (DCCA) loss function that forces the semantic embedding spaces of both views to be maximally correlated. We also detail how the EEG dataset was collected and use it to train our proposed model. We evaluate the learned semantic space in a transfer learning context, by using it as an audio feature extractor in an independent dataset and proxy task: music audio-lyrics cross-modal retrieval. We show that our embedding model outperforms Spotify features and performs comparably to a state-of-the-art embedding model that was trained on 700 times more data. We further discuss improvements to the model that are likely to improve its performance.

Motivation & Objective

• To address the limitation of audio-only models in capturing subjective music semantics by incorporating neural correlates of perception.
• To develop a generic framework that integrates EEG data as a regularizer to improve semantic embedding quality in music representation learning.
• To evaluate the effectiveness of EEG-regularized embeddings in a transfer learning setting for cross-modal retrieval.
• To demonstrate that perceptually grounded embeddings can match or exceed performance of models trained on vastly larger audio-only datasets.

Proposed method

• Designs a 2-view neural network architecture that processes 1.5-second aligned audio and EEG inputs in an end-to-end manner.
• Employs Deep Canonical Correlation Analysis (DCCA) as the loss function to maximize correlation between audio and EEG embedding spaces.
• Segments each audio-EEG recording into 1.5-second chunks for training, with song-level embeddings obtained by averaging chunk-level outputs.
• Uses a 5-fold cross-validation setup with 20 epochs per fold, training on 2.5 hours of audio-EEG data collected from human listeners.
• Applies the trained model as a feature extractor in a proxy task: audio-lyrics cross-modal retrieval.
• Compares performance against Spotify’s handcrafted features and a SOTA model (Choi et al.) trained on 2083+ hours of audio.

Experimental results

Research questions

• RQ1 Can EEG signals serve as an effective regularizer to improve deep learning models of music semantics beyond audio alone?
• RQ2 Does a DCCA-based joint embedding space of audio and EEG yield better transferable representations than audio-only models?
• RQ3 Can a model trained on minimal audio-EEG data (under 3 hours) outperform a SOTA model trained on 700× more data?
• RQ4 How does the inclusion of perceptual brain signals affect the quality and generalization of music semantic embeddings?

Key findings

• The proposed EEG-regularized model achieves an instance-based Mean Reciprocal Rank (MRR) of 24.6% on audio-lyrics retrieval, outperforming Spotify’s 23.4%.
• The model achieves a class-based MRR of 36.2%, surpassing Spotify’s 35.1% and matching the SOTA model (Choi) at 36.5%.
• Despite being trained on less than 3 hours of audio-EEG data, the model performs comparably to a model trained on 2083+ hours of audio.
• The model completes training in approximately 20 minutes on a single GPU, demonstrating high training efficiency.
• Qualitative advantages include fine-grained temporal alignment between audio and EEG and avoidance of fixed semantic taxonomies.
• Future improvements such as residual connections, LSTMs for embedding aggregation, and personalized subject modeling are expected to further enhance performance.