[Paper Review] MASA-TCN: Multi-anchor Space-aware Temporal Convolutional Neural Networks for Continuous and Discrete EEG Emotion Recognition

Yi Ding, Su Zhang+2 more|arXiv (Cornell University)|Aug 30, 2023

EEG and Brain-Computer InterfacesNeuroscience3 citations

TL;DR

This paper proposes MASA-TCN, a unified deep learning model that enhances EEG-based continuous and discrete emotion recognition by integrating space-aware temporal convolutional networks with a multi-anchor attentive fusion block. The model captures spatial, spectral, and temporal patterns simultaneously, achieving state-of-the-art performance on two public EEG datasets for both regression and classification tasks.

ABSTRACT

Emotion recognition using electroencephalogram (EEG) mainly has two scenarios: classification of the discrete labels and regression of the continuously tagged labels. Although many algorithms were proposed for classification tasks, there are only a few methods for regression tasks. For emotion regression, the label is continuous in time. A natural method is to learn the temporal dynamic patterns. In previous studies, long short-term memory (LSTM) and temporal convolutional neural networks (TCN) were utilized to learn the temporal contextual information from feature vectors of EEG. However, the spatial patterns of EEG were not effectively extracted. To enable the spatial learning ability of TCN towards better regression and classification performances, we propose a novel unified model, named MASA-TCN, for EEG emotion regression and classification tasks. The space-aware temporal layer enables TCN to additionally learn from spatial relations among EEG electrodes. Besides, a novel multi-anchor block with attentive fusion is proposed to learn dynamic temporal dependencies. Experiments on two publicly available datasets show MASA-TCN achieves higher results than the state-of-the-art methods for both EEG emotion regression and classification tasks. The code is available at https://github.com/yi-ding-cs/MASA-TCN.

Motivation & Objective

• Address the limited effectiveness of existing methods in capturing spatial patterns among EEG electrodes during emotion recognition.
• Improve temporal modeling for continuous EEG emotion regression by learning dynamic dependencies across multiple time scales.
• Unify continuous emotion regression (CER) and discrete emotion classification (DEC) into a single deep learning framework.
• Enhance model performance by integrating spatial, spectral, and temporal representations through a novel architecture.
• Demonstrate superior performance on public EEG emotion datasets compared to state-of-the-art methods in both CER and DEC tasks.

Proposed method

• Propose a Space-aware Temporal (SAT) layer that enables the TCN to explicitly model spatial relationships between EEG electrodes.
• Introduce a Multi-anchor Attentive Fusion (MAAF) block that captures temporal dynamics at multiple receptive fields using dilated convolutions.
• Apply attentive fusion across multi-scale temporal features to emphasize relevant temporal patterns for emotion prediction.
• Use a shared backbone for both CER and DEC tasks, with a mean fusion layer applied to the regressor output to enable end-to-end classification.
• Train the model using a combination of loss functions, primarily CCC, with potential for future integration of RMSE and PCC for improved detail regression.
• Leverage raw EEG signals directly without hand-crafted features, enabling end-to-end learning of spatial-temporal patterns.

Figure 1: The architecture of our MASA-TCN. There are four main parts of MASA-TCN: feature extraction block, MAAF block, TCN block, and regression/classification block. A sequence of five four-EEG-channel sub-segments is utilized as an example. The k of SAT refers to the length of the kernel in the

Experimental results

Research questions

• RQ1 Can a unified deep learning model effectively handle both continuous and discrete EEG-based emotion recognition tasks?
• RQ2 How does incorporating spatial relationships among EEG electrodes improve performance in emotion recognition compared to models that ignore spatial structure?
• RQ3 To what extent does multi-scale temporal modeling via a multi-anchor block enhance the learning of dynamic emotional states in EEG signals?
• RQ4 Does using the mean of the regressor output as a classification head improve performance in discrete emotion classification compared to standard classifiers?
• RQ5 How do different evaluation metrics (RMSE, PCC, CCC) reflect the true performance of emotion regression models, and which is most reliable for model optimization?

Key findings

• MASA-TCN achieves state-of-the-art results on two public EEG emotion datasets for both continuous emotion regression and discrete emotion classification tasks.
• The inclusion of the space-aware temporal (SAT) layer significantly improves performance by effectively modeling spatial patterns across EEG electrodes.
• The multi-anchor attentive fusion (MAAF) block enables the model to capture complex temporal dynamics at varying time scales, enhancing regression accuracy.
• Using the mean of the regressor output as a classification head yields measurable improvements in F1 score, demonstrating the effectiveness of the unified architecture.
• Evaluation results show that CCC is a more reliable metric than RMSE or PCC for continuous emotion regression, as it captures both correlation and agreement in absolute values.
• The model performs better on positive-valence labels and struggles with sudden amplitude changes, indicating a need for improved regularization in future work.

Figure 2: Space-aware temporal convolutional layer. The SAT has two types of convolutional kernels: context kernels that extract the spectral patterns channel by channel and spatial fusion kernels that learn spatial patterns across all the channels. A context kernel of size (4, 3) is utilized for ex