[Paper Review] DialogueTRM: Exploring the Intra- and Inter-Modal Emotional Behaviors in the Conversation

Yuzhao Mao, Qi Sun+5 more|arXiv (Cornell University)|Oct 15, 2020

Language, Metaphor, and Cognition23 references18 citations

TL;DR

This paper proposes DialogueTRM, a novel multi-modal transformer architecture that models intra-modal context preferences and inter-modal feature interactions for emotion recognition in conversations. By integrating a Hierarchical Transformer for modality-specific context handling and a Multi-Grained Interactive Fusion for neuron- and vector-level cross-modal attention, DialogueTRM achieves state-of-the-art performance on three benchmark datasets, outperforming prior methods significantly.

ABSTRACT

Emotion Recognition in Conversations (ERC) is essential for building empathetic human-machine systems. Existing studies on ERC primarily focus on summarizing the context information in a conversation, however, ignoring the differentiated emotional behaviors within and across different modalities. Designing appropriate strategies that fit the differentiated multi-modal emotional behaviors can produce more accurate emotional predictions. Thus, we propose the DialogueTransformer to explore the differentiated emotional behaviors from the intra- and inter-modal perspectives. For intra-modal, we construct a novel Hierarchical Transformer that can easily switch between sequential and feed-forward structures according to the differentiated context preference within each modality. For inter-modal, we constitute a novel Multi-Grained Interactive Fusion that applies both neuron- and vector-grained feature interactions to learn the differentiated contributions across all modalities. Experimental results show that DialogueTRM outperforms the state-of-the-art by a significant margin on three benchmark datasets.

Motivation & Objective

• To address the gap in existing emotion recognition in conversations (ERC) models that overlook differentiated emotional behaviors within and across modalities.
• To model intra-modal context preferences—where text relies on context while vision and speech are context-free—through a dynamic structural switch in a Hierarchical Transformer.
• To learn modality-specific contributions in multi-modal fusion by enabling both neuron-grained and vector-grained feature interactions.
• To improve emotional prediction accuracy by jointly modeling temporal dependencies within modalities and spatial dependencies across modalities.

Proposed method

• A Hierarchical Transformer (HT) is designed with learnable attention masks to switch between sequential (context-dependent) and feed-forward (context-free) structures based on modality-specific context preferences.
• The HT combines BERT and Transformer components to process modality-specific representations with adaptive context handling.
• A Multi-Grained Interactive Fusion (MGIF) module is proposed, using a multi-modal gate for neuron-grained fusion to align features across different modality spaces.
• The MGIF employs a Transformer encoder to perform vector-grained fusion, learning dynamic attention weights for modality contributions.
• The full DialogueTRM model integrates HT and MGIF modules to jointly optimize intra- and inter-modal emotional behavior modeling.
• The model is trained end-to-end on three multi-modal ERC benchmarks using cross-entropy loss with early stopping.

Experimental results

Research questions

• RQ1 How do emotional expressions within individual modalities (text, audio, visual) differ in their dependence on conversational context?
• RQ2 What is the optimal structural design for modeling modality-specific context preferences in multi-modal emotion recognition?
• RQ3 How can multi-modal representations with differing feature spaces be effectively fused to capture modality-specific contributions?
• RQ4 To what extent does multi-grained (neuron- and vector-level) fusion improve emotion recognition performance compared to single-grained or non-interactive fusion?

Key findings

• DialogueTRM achieves state-of-the-art performance on three benchmark datasets: IEMOCAP, MELD, and DailyDialog, outperforming prior SOTA methods.
• The MGIF module achieves a 3× higher F1 score gain compared to MulT, demonstrating the effectiveness of multi-grained fusion.
• Textual modality shows significantly better performance under context-dependent settings, while visual and acoustic modalities perform best under context-free settings.
• The ablation study confirms that both neuron-grained fusion (via multi-modal gate) and vector-grained fusion (via Transformer) are essential, with the latter outperforming linear concatenation.
• In cases where text alone misclassifies emotions, multi-modal fusion corrects up to 85.41% of 'neutral' and 70.45% of 'frustrated' predictions, especially for negative emotions masked by neutral language.
• For the utterance 'yeah.', accuracy improves from 43.48% (text-only) to 65.22% with context and 73.91% with multi-modal features, highlighting the value of multi-modal context.