[Paper Review] HADUA: Hierarchical Attention and Dynamic Uniform Alignment for Robust Cross-Subject Emotion Recognition

Jiahao Tang, Youjun Li+5 more|arXiv (Cornell University)|Jan 29, 2026

Emotion and Mood Recognition0 citations

TL;DR

HADUA integrates hierarchical attention-based multimodal fusion with confidence-aware pseudo-label weighting and uniform class-wise alignment to achieve robust cross-subject emotion recognition from EEG and eye movement signals.

ABSTRACT

Robust cross-subject emotion recognition from multimodal physiological signals remains a challenging problem, primarily due to modality heterogeneity and inter-subject distribution shift. To tackle these challenges, we propose a novel adaptive learning framework named Hierarchical Attention and Dynamic Uniform Alignment (HADUA). Our approach unifies the learning of multimodal representations with domain adaptation. First, we design a hierarchical attention module that explicitly models intra-modal temporal dynamics and inter-modal semantic interactions (e.g., between electroencephalogram(EEG) and eye movement(EM)), yielding discriminative and semantically coherent fused features. Second, to overcome the noise inherent in pseudo-labels during adaptation, we introduce a confidence-aware Gaussian weighting scheme that smooths the supervision from target-domain samples by down-weighting uncertain instances. Third, a uniform alignment loss is employed to regularize the distribution of pseudo-labels across classes, thereby mitigating imbalance and stabilizing conditional distribution matching. Extensive experiments on multiple cross-subject emotion recognition benchmarks show that HADUA consistently surpasses existing state-of-the-art methods in both accuracy and robustness, validating its effectiveness in handling modality gaps, noisy pseudo-labels, and class imbalance. Taken together, these contributions offer a practical and generalizable solution for building robust cross-subject affective computing systems.

Motivation & Objective

• Address cross-subject distribution shift in multimodal emotion recognition.
• Model intra-modal temporal dynamics and inter-modal semantic interactions for EEG and EM.
• Mitigate noisy pseudo-labels in target domain via confidence-aware weighting.
• Balance class contributions during conditional distribution alignment to handle imbalance.

Proposed method

• Proposes a dual-branch feature extractor for EEG and eye-movement signals with hierarchical attention (self-attention per modality and uni-directional cross-attention EEG→Eye).
• Uses MMD to align marginal distributions and CMMD for conditional distribution alignment based on target pseudo-labels.
• Introduces Soft Gaussian Weighting to softly weight target pseudo-labels by prediction confidence with EMA-updated statistics.
• Implements Uniform Alignment to softly rebalance pseudo-label class distributions, controlled by a scheduling factor and temperature parameter.
• Trains end-to-end with a closed-loop framework where multimodal representation learning, pseudo-label optimization, and distribution alignment interact.
• Applies differential entropy-based EEG features and comprehensive eye-movement features, followed by Transformer-style attention for fusion.

Experimental results

Research questions

• RQ1 Can hierarchical attention effectively model intra-modal structure and inter-modal semantics for EEG and EM signals in cross-subject settings?
• RQ2 Does a confidence-aware weighting of target pseudo-labels reduce the impact of label noise during domain alignment?
• RQ3 Can uniform class-wise pseudo-label balancing improve CPD alignment and overall generalization under cross-subject shifts?
• RQ4 What is the impact of combining MPD (MMD) and CPD (CMMD) alignment with pseudo-label optimization on cross-subject robustness?

Key findings

• HADUA consistently surpasses state-of-the-art methods in accuracy and robustness across cross-subject benchmarks.
• Hierarchical attention captures intra-modal temporal and inter-modal semantic interactions, improving discriminative fused features.
• Soft Gaussian weighting stabilizes training by down-weighting low-confidence pseudo-labels without discarding data.
• Uniform Alignment mitigates class imbalance in pseudo-labels, improving conditional distribution matching.