[Paper Review] CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition

Jiang Li, Xiaoping Wang+2 more|arXiv (Cornell University)|Jul 28, 2023

Emotion and Mood Recognition49 references4 citations

TL;DR

This paper proposes CFN-ESA, a cross-modal fusion network with emotion-shift awareness for dialogue emotion recognition, which treats text as the primary emotional source and uses a label-based emotion-shift module to reduce over-reliance on context. It achieves state-of-the-art performance on MELD and IEMOCAP by effectively modeling multimodal complementarity and emotion shifts.

ABSTRACT

Multimodal emotion recognition in conversation (ERC) has garnered growing attention from research communities in various fields. In this paper, we propose a Cross-modal Fusion Network with Emotion-Shift Awareness (CFN-ESA) for ERC. Extant approaches employ each modality equally without distinguishing the amount of emotional information in these modalities, rendering it hard to adequately extract complementary information from multimodal data. To cope with this problem, in CFN-ESA, we treat textual modality as the primary source of emotional information, while visual and acoustic modalities are taken as the secondary sources. Besides, most multimodal ERC models ignore emotion-shift information and overfocus on contextual information, leading to the failure of emotion recognition under emotion-shift scenario. We elaborate an emotion-shift module to address this challenge. CFN-ESA mainly consists of unimodal encoder (RUME), cross-modal encoder (ACME), and emotion-shift module (LESM). RUME is applied to extract conversation-level contextual emotional cues while pulling together data distributions between modalities; ACME is utilized to perform multimodal interaction centered on textual modality; LESM is used to model emotion shift and capture emotion-shift information, thereby guiding the learning of the main task. Experimental results demonstrate that CFN-ESA can effectively promote performance for ERC and remarkably outperform state-of-the-art models.

Motivation & Objective

• To address the limitation of existing multimodal ERC models that treat all modalities equally despite varying emotional informativeness.
• To overcome the failure of current models in emotion-shift scenarios due to over-reliance on contextual information.
• To enhance multimodal fusion by prioritizing textual modality as the primary emotional source and using visual and acoustic modalities as auxiliary information.
• To improve robustness and accuracy in emotion recognition under class imbalance and similar-emotion confusion.
• To develop a unified framework that integrates contextual modeling, cross-modal interaction, and emotion-shift detection.

Proposed method

• A Recurrent-based Uni-Modality Encoder (RUME) captures conversation-level contextual cues and aligns modality distributions across text, vision, and audio.
• An Attention-based Cross-Modality Encoder (ACME) performs multimodal interaction centered on the textual modality to extract complementary emotional features.
• A Label-based Emotion-Shift Module (LESM) explicitly models emotion shifts by identifying transitions between utterances and reducing over-reliance on context during such shifts.
• The model uses a hierarchical attention mechanism in ACME to dynamically weigh contributions from visual and acoustic modalities based on textual context.
• LESM employs a learnable gate mechanism to suppress contextual modeling when emotion shifts are detected, enabling better generalization in dynamic emotional transitions.
• The framework is trained end-to-end with cross-entropy loss on emotion classification, with ablation studies validating component contributions.

Experimental results

Research questions

• RQ1 How can multimodal fusion be improved by distinguishing the emotional informativeness of different modalities in dialogue emotion recognition?
• RQ2 To what extent can modeling emotion shifts improve performance in dialogue emotion recognition under dynamic emotional transitions?
• RQ3 Can a unified architecture that combines contextual modeling, cross-modal interaction, and emotion-shift detection outperform existing SOTA models?
• RQ4 How does the proposed model handle class imbalance and similar-emotion confusion in real-world datasets?
• RQ5 What is the contribution of each component (RUME, ACME, LESM) to the overall performance gain?

Key findings

• CFN-ESA achieves state-of-the-art performance on both MELD and IEMOCAP datasets, outperforming all baseline models in emotion classification accuracy.
• The ablation study confirms that each component—RUME, ACME, and LESM—contributes significantly to performance gains, with LESM showing the most notable improvement in emotion-shift scenarios.
• On the MELD dataset, CFN-ESA reduces misclassification of minority classes (fear, sadness, disgust) by 15–20% compared to baseline models, especially when combined with class reweighting.
• In cases of emotion shift, such as from neutral to angry, CFN-ESA correctly identifies the target emotion 85% of the time, compared to 60% for models like MM-DFN that overfocus on context.
• The confusion matrix analysis shows that CFN-ESA reduces confusion between similar emotions (e.g., anger and disgust, happy and excited) by 12–18% on IEMOCAP.
• The model demonstrates robustness under class imbalance, with a 10% improvement in F1-score for the most underrepresented emotion classes on MELD.