[Paper Review] EEG2Mel: Reconstructing Sound from Brain Responses to Music

Adolfo G. Ramirez-Aristizabal, Chris Kello|arXiv (Cornell University)|Jul 28, 2022

EEG and Brain-Computer Interfaces4 citations

TL;DR

This paper proposes EEG2Mel, a deep learning framework that reconstructs perceptible music from EEG signals using time-aligned one-second EEG and spectrogram pairs. By treating EEG power spectra and mel-spectrograms as 2D images, a CNN regressor achieves 85% listener identification accuracy in a two-alternative match-to-sample task, demonstrating high-fidelity auditory reconstruction from brain activity without extensive preprocessing.

ABSTRACT

Information retrieval from brain responses to auditory and visual stimuli has shown success through classification of song names and image classes presented to participants while recording EEG signals. Information retrieval in the form of reconstructing auditory stimuli has also shown some success, but here we improve on previous methods by reconstructing music stimuli well enough to be perceived and identified independently. Furthermore, deep learning models were trained on time-aligned music stimuli spectrum for each corresponding one-second window of EEG recording, which greatly reduces feature extraction steps needed when compared to prior studies. The NMED-Tempo and NMED-Hindi datasets of participants passively listening to full length songs were used to train and validate Convolutional Neural Network (CNN) regressors. The efficacy of raw voltage versus power spectrum inputs and linear versus mel spectrogram outputs were tested, and all inputs and outputs were converted into 2D images. The quality of reconstructed spectrograms was assessed by training classifiers which showed 81% accuracy for mel-spectrograms and 72% for linear spectrograms (10% chance accuracy). Lastly, reconstructions of auditory music stimuli were discriminated by listeners at an 85% success rate (50% chance) in a two-alternative match-to-sample task.

Motivation & Objective

• To develop a method for reconstructing naturalistic, full-length music stimuli from passive EEG responses during listening.
• To improve upon prior work by enabling perceptible and identifiable music reconstruction rather than just classification or low-fidelity signal retrieval.
• To eliminate the need for heavy feature extraction by directly using EEG power spectra as input to deep learning models.
• To validate the perceptual quality of reconstructed music through behavioral listening tasks with human participants.
• To demonstrate the feasibility of applying computer vision techniques to time-series neural and audio data for cross-domain signal reconstruction.

Proposed method

• EEG signals from 63 channels were converted into 2D power spectrum images for each one-second window, aligned with corresponding music spectrograms.
• A deep CNN regressor was trained to map EEG power spectrum images to mel-spectrogram images using the NMED-Tempo and NMED-Hindi datasets.
• The model architecture included multiple 2D convolutional layers with batch normalization, max pooling, and fully connected layers followed by a reshape operation to reconstruct the spectrogram.
• Spectrogram inversion to waveform was performed using Griffin-Lim algorithm after denormalization using song-specific reference dB values to reduce perceptual artifacts.
• Input data were normalized using min-max scaling, and output spectrograms were denormalized using a song-specific reference dB value to improve audio quality.
• The reconstruction quality was evaluated via a two-alternative forced-choice listening task with 16 participants, assessing perceptual similarity to original music.

Experimental results

Research questions

• RQ1 Can EEG signals be used to reconstruct full-length, perceptually identifiable music stimuli without relying on repeated stimulus averaging or short bursts?
• RQ2 To what extent can a deep learning model trained on EEG power spectra reconstruct mel-spectrograms well enough to produce listenable and recognizable music?
• RQ3 Does representing EEG and spectrogram data as 2D images enable better performance and simpler training compared to traditional time-series modeling?
• RQ4 How does the choice of input (raw voltage vs. power spectrum) and output (linear vs. mel-spectrogram) affect reconstruction quality?
• RQ5 Can listeners reliably identify reconstructed music as matching the original stimulus in a perceptual discrimination task?

Key findings

• The model achieved 81% accuracy in classifying reconstructed mel-spectrograms using a downstream classifier, significantly above the 10% chance baseline.
• Linear spectrogram reconstruction achieved 72% classification accuracy, also above chance, indicating partial but weaker signal recovery.
• Listeners correctly identified the original music in 85% of trials during a two-alternative match-to-sample task, well above the 50% chance level.
• The use of mel-spectrogram outputs led to higher perceptual quality than linear spectrograms, supporting the choice of mel-scale for auditory reconstruction.
• The reconstruction quality was improved by using song-specific reference dB values during denormalization, reducing perceptual artifacts.
• The approach successfully preserved temporal dynamics and frequency content across the full music duration, enabling coherent and identifiable audio output.