[Paper Review] EmT: A Novel Transformer for Generalized Cross-subject EEG Emotion Recognition

Yi Ding, Chengxuan Tong+5 more|arXiv (Cornell University)|Jun 26, 2024

EEG and Brain-Computer Interfaces4 citations

TL;DR

EmT is a novel graph-transformer model that enhances generalized cross-subject EEG emotion recognition by modeling long-term temporal context and multi-view brain connectivity. It uses a temporal graph construction module to create dynamic EEG feature graphs, a residual multi-view pyramid GCN (RMPG) to learn diverse spatial connections, and a temporal contextual transformer (TCT) with dual token mixers to capture sequential dependencies, achieving state-of-the-art performance on both classification and regression tasks across four public datasets.

ABSTRACT

Integrating prior knowledge of neurophysiology into neural network architecture enhances the performance of emotion decoding. While numerous techniques emphasize learning spatial and short-term temporal patterns, there has been limited emphasis on capturing the vital long-term contextual information associated with emotional cognitive processes. In order to address this discrepancy, we introduce a novel transformer model called emotion transformer (EmT). EmT is designed to excel in both generalized cross-subject EEG emotion classification and regression tasks. In EmT, EEG signals are transformed into a temporal graph format, creating a sequence of EEG feature graphs using a temporal graph construction module (TGC). A novel residual multi-view pyramid GCN module (RMPG) is then proposed to learn dynamic graph representations for each EEG feature graph within the series, and the learned representations of each graph are fused into one token. Furthermore, we design a temporal contextual transformer module (TCT) with two types of token mixers to learn the temporal contextual information. Finally, the task-specific output module (TSO) generates the desired outputs. Experiments on four publicly available datasets show that EmT achieves higher results than the baseline methods for both EEG emotion classification and regression tasks. The code is available at https://github.com/yi-ding-cs/EmT.

Motivation & Objective

• To address the lack of long-term contextual modeling in existing EEG emotion recognition models.
• To integrate neurophysiological priors such as dynamic brain connectivity and hemispheric asymmetry into deep learning architectures.
• To develop a unified framework that performs robustly on both classification and regression tasks in cross-subject EEG emotion recognition.
• To enable the model to learn multiple, diverse brain connectivity patterns corresponding to different cognitive processes during emotional stimulation.

Proposed method

• EEG signals are transformed into temporal graphs via a Temporal Graph Construction (TGC) module, where each segment is split into sub-segments and frequency-band features are extracted.
• A Residual Multi-view Pyramid GCN (RMPG) module applies multiple parallel GCNs with learnable adjacency matrices to capture multi-level, multi-view spatial representations across all EEG channels.
• The RMPG outputs are fused via mean pooling to form a single token per temporal graph, creating a sequence of token embeddings.
• A Temporal Contextual Transformer (TCT) module with two specialized token mixer designs (TCT-Clas and TCT-Regr) processes the sequence to model long-range temporal dependencies.
• Task-specific output modules (TSO) generate predictions: a classification head for emotion class labels and a regression head for continuous valence/arousal scores.
• The model is trained end-to-end on four public EEG datasets, with ablation studies to validate component contributions.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture effectively model long-term temporal contextual information in EEG signals for emotion recognition?
• RQ2 How does learning multiple, diverse brain connectivity patterns (via multiple GCNs with learnable adjacency matrices) improve performance in cross-subject EEG emotion recognition?
• RQ3 Does the integration of neurophysiologically informed spatial and temporal modeling lead to better generalization across unseen subjects?
• RQ4 How do different EEG feature types (e.g., rPSD, DE, PSD) impact the performance of the EmT model in classification and regression tasks?
• RQ5 To what extent do the number of TCT blocks influence performance on classification versus regression tasks?

Key findings

• EmT achieves state-of-the-art performance on both EEG emotion classification and regression tasks across four public datasets, with accuracy reaching 0.802 and F1 score of 0.821 on SEED.
• The use of relative power spectral density (rPSD) as input features yields the best performance, outperforming both differential entropy (DE) and power spectral density (PSD).
• Increasing the number of TCT blocks significantly improves classification performance (accuracy from 0.780 to 0.802), but has minimal impact on regression metrics (RMSE ~0.06, PCC ~0.48, CCC ~0.39).
• The model learns two distinct, neurophysiologically meaningful brain connectivity patterns: one involving frontal-parietal-temporal regions linked to attention, and another involving frontal-temporal-parietal-occipital networks related to emotional and visual processing.
• Visualization of learned adjacency matrices confirms that EmT captures task-specific connectivity patterns, such as strong F6-POz and FC4-P7 connections during emotional stimulation.
• The residual multi-view pyramid GCN (RMPG) effectively captures diverse, multi-level graph representations by learning independent adjacency matrices across multiple GCN branches.