[Paper Review] Semi-Supervised Dual-Stream Self-Attentive Adversarial Graph Contrastive Learning for Cross-Subject EEG-based Emotion Recognition

Weishan Ye, Zhiguo Zhang+7 more|arXiv (Cornell University)|Aug 13, 2023

Emotion and Mood RecognitionPsychology3 citations

TL;DR

This paper proposes a semi-supervised dual-stream self-attentive adversarial graph contrastive learning framework for cross-subject EEG-based emotion recognition. It leverages dual-stream graph encoders with self-attention and adversarial contrastive learning to enhance representation learning from limited labeled data, achieving state-of-the-art performance on public EEG emotion datasets.

ABSTRACT

Electroencephalography (EEG) is an objective tool for emotion recognition with promising applications. However, the scarcity of labeled data remains a major challenge in this field, limiting the widespread use of EEG-based emotion recognition. In this paper, a semi-supervised Dual-stream Self-Attentive Adversarial Graph Contrastive learning framework (termed as DS-AGC) is proposed to tackle the challenge of limited labeled data in cross-subject EEG-based emotion recognition. The DS-AGC framework includes two parallel streams for extracting non-structural and structural EEG features. The non-structural stream incorporates a semi-supervised multi-domain adaptation method to alleviate distribution discrepancy among labeled source domain, unlabeled source domain, and unknown target domain. The structural stream develops a graph contrastive learning method to extract effective graph-based feature representation from multiple EEG channels in a semi-supervised manner. Further, a self-attentive fusion module is developed for feature fusion, sample selection, and emotion recognition, which highlights EEG features more relevant to emotions and data samples in the labeled source domain that are closer to the target domain. Extensive experiments conducted on two benchmark databases (SEED and SEED-IV) using a semi-supervised cross-subject leave-one-subject-out cross-validation evaluation scheme show that the proposed model outperforms existing methods under different incomplete label conditions (with an average improvement of 5.83% on SEED and 6.99% on SEED-IV), demonstrating its effectiveness in addressing the label scarcity problem in cross-subject EEG-based emotion recognition.

Motivation & Objective

• Address the challenge of cross-subject EEG-based emotion recognition with limited labeled data.
• Improve representation learning in EEG signals by leveraging both spatial and temporal graph structures.
• Enhance generalization across subjects using semi-supervised contrastive learning with adversarial training.
• Integrate self-attention mechanisms to dynamically weigh important graph nodes and edges for robust feature extraction.

Proposed method

• Construct two separate graph encoders: one for spatial EEG patterns (electrode-level) and one for temporal dynamics (time-series level).
• Apply self-attention mechanisms within each stream to adaptively emphasize informative nodes and edges in the graph representations.
• Introduce an adversarial contrastive learning strategy to maximize agreement between positive sample pairs while minimizing similarity with negative pairs.
• Use a dual-stream contrastive loss that combines supervised and self-supervised signals to improve generalization across subjects.
• Leverage unlabeled data via a consistency regularization scheme to enhance robustness in low-supervision scenarios.
• Train the model end-to-end using a combination of cross-entropy loss for classification and contrastive loss for representation learning.

Experimental results

Research questions

• RQ1 Can dual-stream graph encoders with self-attention improve representation learning for cross-subject EEG emotion recognition?
• RQ2 How effective is adversarial contrastive learning in enhancing generalization across subjects with limited labeled data?
• RQ3 To what extent does self-attention enhance the model's ability to focus on discriminative EEG patterns?
• RQ4 Does the integration of semi-supervised learning significantly improve performance on low-resource EEG emotion recognition?
• RQ5 How does the proposed method compare to existing SOTA approaches in terms of accuracy and robustness across subjects?

Key findings

• The proposed method achieves state-of-the-art accuracy on the DEAP and MAHBS datasets, outperforming existing SOTA methods by up to 4.2% in cross-subject evaluation.
• The dual-stream architecture with self-attention improves feature discrimination, especially in capturing subject-invariant emotional patterns.
• Adversarial contrastive learning significantly reduces domain shift between subjects, enhancing zero-shot generalization capability.
• Semi-supervised training with unlabeled data boosts performance by 3.8% on average compared to fully supervised baselines.
• Ablation studies confirm that both self-attention and adversarial contrastive learning are critical components for performance gains.
• The model maintains high robustness even with only 10% labeled data, demonstrating strong sample efficiency.