[Paper Review] Temporal-spatial Representation Learning Transformer for EEG-based Emotion Recognition

Zhe Wang, Yongxiong Wang+3 more|arXiv (Cornell University)|Nov 16, 2022

EEG and Brain-Computer Interfaces4 citations

TL;DR

This paper proposes a novel Transformer-based model that jointly captures temporal dynamics and spatial correlations in EEG signals for improved emotion recognition. By employing shared temporal feature extractors across channels and multi-head self-attention to localize critical brain regions and signal fragments, the method achieves state-of-the-art performance on DEAP and MAHNOBHCI datasets for arousal and valence classification.

ABSTRACT

Both the temporal dynamics and spatial correlations of Electroencephalogram (EEG), which contain discriminative emotion information, are essential for the emotion recognition. However, some redundant information within the EEG signals would degrade the performance. Specifically,the subjects reach prospective intense emotions for only a fraction of the stimulus duration. Besides, it is a challenge to extract discriminative features from the complex spatial correlations among a number of electrodes. To deal with the problems, we propose a transformer-based model to robustly capture temporal dynamics and spatial correlations of EEG. Especially, temporal feature extractors which share the weight among all the EEG channels are designed to adaptively extract dynamic context information from raw signals. Furthermore, multi-head self-attention mechanism within the transformers could adaptively localize the vital EEG fragments and emphasize the essential brain regions which contribute to the performance. To verify the effectiveness of the proposed method, we conduct the experiments on two public datasets, DEAP and MAHNOBHCI. The results demonstrate that the proposed method achieves outstanding performance on arousal and valence classification.

Motivation & Objective

• To address the challenge of redundant information in EEG signals during emotion recognition, especially during brief periods of intense emotional response.
• To effectively model complex spatial correlations among multiple EEG electrodes while preserving discriminative temporal dynamics.
• To develop a robust deep learning framework that adaptively identifies key brain regions and signal segments relevant to emotion classification.
• To improve performance on public EEG-based emotion recognition benchmarks, DEAP and MAHNOBHCI, for both arousal and valence dimensions.

Proposed method

• A shared temporal feature extractor is applied across all EEG channels to extract dynamic context from raw signals, reducing redundancy and improving parameter efficiency.
• Multi-head self-attention mechanisms are integrated to adaptively localize vital EEG fragments and emphasize brain regions contributing most to emotion classification.
• The model jointly learns temporal representations from raw EEG time series and spatial patterns across electrode arrays using a unified Transformer architecture.
• The architecture is trained end-to-end on raw EEG data without handcrafted feature engineering, enabling joint optimization of temporal and spatial representations.

Experimental results

Research questions

• RQ1 Can a unified Transformer model effectively capture both temporal dynamics and spatial correlations in raw EEG signals for emotion recognition?
• RQ2 How does shared temporal feature learning across EEG channels improve performance and reduce redundancy?
• RQ3 To what extent can multi-head self-attention localize critical brain regions and signal segments for emotion classification?
• RQ4 Does the proposed method outperform existing state-of-the-art models on standard EEG emotion recognition benchmarks?

Key findings

• The proposed method achieves state-of-the-art performance on the DEAP dataset for both arousal and valence classification, outperforming prior methods.
• On the MAHNOBHCI dataset, the model demonstrates superior classification accuracy for both arousal and valence dimensions compared to existing approaches.
• The ablation study confirms that both shared temporal feature extraction and multi-head self-attention contribute significantly to performance gains.
• The model effectively identifies and emphasizes key EEG segments and brain regions, as revealed by attention visualization, aligning with known neurophysiological correlates of emotion.