[Paper Review] Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shifts

Harsh Agarwal, Keshav Bansal+2 more|arXiv (Cornell University)|Dec 3, 2021

Emotion and Mood Recognition12 citations

TL;DR

This paper proposes a multimodal emotion recognition model enhanced with a modular emotion-shift component that improves performance on conversations with abrupt emotional transitions. By using a Siamese network to detect shifts between consecutive utterances and integrating it with GRU-based fusion, the model outperforms existing methods on MOSEI and IEMOCAP, especially in predicting emotion shifts, with a 20% improvement on negative-to-positive transitions.

ABSTRACT

Emotion Recognition in Conversations (ERC) is an important and active research area. Recent work has shown the benefits of using multiple modalities (e.g., text, audio, and video) for the ERC task. In a conversation, participants tend to maintain a particular emotional state unless some stimuli evokes a change. There is a continuous ebb and flow of emotions in a conversation. Inspired by this observation, we propose a multimodal ERC model and augment it with an emotion-shift component that improves performance. The proposed emotion-shift component is modular and can be added to any existing multimodal ERC model (with a few modifications). We experiment with different variants of the model, and results show that the inclusion of emotion shift signal helps the model to outperform existing models for ERC on MOSEI and IEMOCAP datasets.

Motivation & Objective

• To improve multimodal emotion recognition in conversations by explicitly modeling emotional shifts between utterances.
• To address the limitation of existing sequential models in handling sudden emotion transitions, especially from positive to negative sentiment.
• To design a modular, pretrainable component that can be integrated into any existing multimodal ERC model with minimal modifications.
• To analyze how GRU gates respond during emotion shifts and validate the component’s impact on information gating.
• To enable real-time applicability by relying only on two consecutive utterances for shift detection.

Proposed method

• A multimodal GRU-based model fuses text, audio, and video features using early fusion (concatenation of modality embeddings).
• An independent emotion-shift component uses a Siamese network to compare consecutive utterances and predict emotional transitions.
• The Siamese network is trained on triplet loss to learn discriminative representations of emotional change, using only textual modality for initial implementation.
• The emotion-shift signal is used to modulate GRU reset and update gates, allowing the model to suppress outdated emotional states during shifts.
• The architecture is modular, enabling integration into existing models with minor architectural changes and fine-tuning.
• Qualitative analysis involves visualizing GRU gate activations to verify that the emotion-shift component learns to reduce reliance on past states during shifts.

Experimental results

Research questions

• RQ1 Can modeling emotional shifts between consecutive utterances improve multimodal emotion recognition performance in conversations?
• RQ2 How does the inclusion of an emotion-shift component affect the prediction of utterances with abrupt sentiment transitions?
• RQ3 To what extent does the emotion-shift module influence GRU gate behavior during emotional transitions?
• RQ4 Can the emotion-shift component be effectively applied to both unimodal and multimodal emotion recognition systems?
• RQ5 Is the emotion-shift component suitable for real-time emotion recognition systems that process dialogues sequentially?

Key findings

• The proposed model outperforms existing state-of-the-art models on both MOSEI and IEMOCAP datasets, particularly in detecting emotion shifts.
• The emotion-shift component leads to a 20% improvement in F1-score for negative-to-positive emotion transitions and a 4% improvement for positive-to-negative transitions compared to Multilogue-Net.
• The Siamese network-based emotion-shift component achieves higher accuracy (82.66%) when using only the text modality, outperforming multimodal fusion (82.20%) in shift detection.
• Qualitative analysis shows that the emotion-shift component successfully reduces GRU reset gate activation during emotional shifts, indicating effective suppression of past emotional states.
• The model’s performance is significantly enhanced on utterances with emotional shifts, demonstrating the component’s effectiveness in handling dynamic emotional transitions.
• The modular design allows the emotion-shift component to be reused across different models and applied in real-time settings using only two consecutive utterances.