[Paper Review] 4D Attention-based Neural Network for EEG Emotion Recognition

Xiao Guo-wen, Mengwen Ye+3 more|arXiv (Cornell University)|Jan 14, 2021

EEG and Brain-Computer Interfaces4 citations

TL;DR

This paper proposes a 4D attention-based neural network (4D-aNN) for EEG emotion recognition, transforming raw EEG signals into 4D spatial-spectral-temporal representations and leveraging spectral, spatial, and temporal attention mechanisms within a CNN-LSTM architecture. The model achieves state-of-the-art performance on the SEED dataset under intra-subject evaluation, demonstrating the effectiveness of multi-domain attention in capturing critical EEG features.

ABSTRACT

Electroencephalograph (EEG) emotion recognition is a significant task in the brain-computer interface field. Although many deep learning methods are proposed recently, it is still challenging to make full use of the information contained in different domains of EEG signals. In this paper, we present a novel method, called four-dimensional attention-based neural network (4D-aNN) for EEG emotion recognition. First, raw EEG signals are transformed into 4D spatial-spectral-temporal representations. Then, the proposed 4D-aNN adopts spectral and spatial attention mechanisms to adaptively assign the weights of different brain regions and frequency bands, and a convolutional neural network (CNN) is utilized to deal with the spectral and spatial information of the 4D representations. Moreover, a temporal attention mechanism is integrated into a bidirectional Long Short-Term Memory (LSTM) to explore temporal dependencies of the 4D representations. Our model achieves state-of-the-art performance on the SEED dataset under intra-subject splitting. The experimental results have shown the effectiveness of the attention mechanisms in different domains for EEG emotion recognition.

Motivation & Objective

• To address the challenge of effectively utilizing multi-domain information (spatial, spectral, temporal) in EEG signals for emotion recognition.
• To develop a unified deep learning framework that captures complex spatiotemporal and spectral dynamics in EEG data.
• To improve state-of-the-art performance in intra-subject EEG emotion recognition by leveraging attention mechanisms across four dimensions.
• To validate the effectiveness of attention mechanisms in enhancing model interpretability and performance in EEG-based affective computing.

Proposed method

• Transforming raw EEG signals into 4D spatial-spectral-temporal representations to preserve multi-domain information.
• Applying spectral and spatial attention mechanisms to dynamically assign importance to different frequency bands and brain regions.
• Using a 3D convolutional neural network (CNN) to extract hierarchical spatial-spectral features from the 4D representations.
• Integrating a bidirectional Long Short-Term Memory (LSTM) with a temporal attention mechanism to model sequential dependencies across time.
• Combining the outputs of the attention-modulated CNN and temporal LSTM for final emotion classification.
• Training the end-to-end 4D-aNN model using supervised learning with cross-entropy loss on the SEED dataset.

Experimental results

Research questions

• RQ1 Can attention mechanisms effectively capture relevant spectral, spatial, and temporal patterns in EEG signals for emotion recognition?
• RQ2 How does the integration of 4D representations improve performance compared to 2D or 3D representations in EEG-based emotion recognition?
• RQ3 To what extent do spectral and spatial attention mechanisms enhance feature learning and model interpretability in EEG data?
• RQ4 Does the proposed 4D-aNN outperform existing state-of-the-art models under intra-subject evaluation on the SEED dataset?
• RQ5 How robust is the model to inter-individual variability in EEG patterns during emotion classification?

Key findings

• The 4D-aNN achieved state-of-the-art performance on the SEED dataset under intra-subject cross-validation, demonstrating superior generalization and accuracy.
• The integration of spectral, spatial, and temporal attention mechanisms significantly improved model performance compared to baseline models without attention.
• The attention mechanisms effectively highlighted discriminative brain regions and frequency bands associated with emotional states, enhancing model interpretability.
• The 3D-CNN component successfully captured complex spatial-spectral patterns from the 4D representations, contributing to high feature discrimination.
• The bidirectional LSTM with temporal attention effectively modeled long-range temporal dependencies in EEG sequences, improving sequential context understanding.
• The ablation study confirmed that each attention mechanism (spectral, spatial, temporal) contributed uniquely to the overall performance gain.