[Paper Review] GA2MIF: Graph and Attention Based Two-Stage Multi-Source Information Fusion for Conversational Emotion Detection

Jiang Li, Xiaoping Wang+2 more|arXiv (Cornell University)|Jul 25, 2022

Sentiment Analysis and Opinion Mining48 references4 citations

TL;DR

This paper proposes GA2MIF, a two-stage multimodal fusion framework for conversational emotion detection that uses multi-head directed graph attention (MDGAT) for intra-modal context modeling and multi-head pairwise cross-modal attention (MPCAT) for inter-modal interaction. By leveraging context windows to reduce redundant connections and avoiding heterogeneous graph input issues, GA2MIF achieves a state-of-the-art weighted F1 score of 70.00% on the IEMOCAP dataset, outperforming prior SOTA models significantly.

ABSTRACT

Multimodal Emotion Recognition in Conversation (ERC) plays an influential role in the field of human-computer interaction and conversational robotics since it can motivate machines to provide empathetic services. Multimodal data modeling is an up-and-coming research area in recent years, which is inspired by human capability to integrate multiple senses. Several graph-based approaches claim to capture interactive information between modalities, but the heterogeneity of multimodal data makes these methods prohibit optimal solutions. In this work, we introduce a multimodal fusion approach named Graph and Attention based Two-stage Multi-source Information Fusion (GA2MIF) for emotion detection in conversation. Our proposed method circumvents the problem of taking heterogeneous graph as input to the model while eliminating complex redundant connections in the construction of graph. GA2MIF focuses on contextual modeling and cross-modal modeling through leveraging Multi-head Directed Graph ATtention networks (MDGATs) and Multi-head Pairwise Cross-modal ATtention networks (MPCATs), respectively. Extensive experiments on two public datasets (i.e., IEMOCAP and MELD) demonstrate that the proposed GA2MIF has the capacity to validly capture intra-modal long-range contextual information and inter-modal complementary information, as well as outperforms the prevalent State-Of-The-Art (SOTA) models by a remarkable margin.

Motivation & Objective

• Address the limitations of existing multimodal conversational emotion recognition models that struggle with heterogeneous modality inputs and redundant graph connections.
• Improve emotion detection accuracy by effectively modeling long-range contextual dependencies within each modality and complementary information across modalities.
• Overcome the shortcomings of prior graph-based models like MMGCN, which treat all modalities as homogeneous nodes, violating GNN assumptions.
• Reduce misclassification of similar emotions (e.g., Sad vs Frustrated) through refined attention mechanisms and structured fusion.
• Demonstrate superior performance across both two- and three-modality settings on public benchmarks IEMOCAP and MELD.

Proposed method

• Employ Multi-head Directed Graph ATtention networks (MDGATs) to model intra-modal long-range dependencies in text, audio, and visual modalities separately.
• Introduce a context window mechanism to construct sparse, meaningful edges between utterances instead of using fully connected graphs, reducing redundancy.
• Apply Multi-head Pairwise Cross-modal ATtention networks (MPCATs) to capture complementary interactions between different modalities at the utterance level.
• Design a novel Sum-Product update function for MDGATs to enhance feature aggregation in the graph attention mechanism.
• Implement a two-stage fusion strategy: first modeling intra-modal context via MDGATs, then fusing modalities through MPCATs for cross-modal understanding.
• Use modality-specific embeddings as input, with attention mechanisms dynamically weighting relevant features across time and modalities.

Experimental results

Research questions

• RQ1 Can a two-stage graph and attention-based fusion framework effectively model intra-modal long-range dependencies in multimodal conversation data?
• RQ2 How does the use of context windows for edge construction compare to fully connected graphs in reducing redundancy while preserving essential contextual relationships?
• RQ3 To what extent can MDGATs and MPCATs jointly improve cross-modal interaction and emotion recognition performance compared to homogeneous graph-based models?
• RQ4 Does the proposed GA2MIF model reduce misclassification of similar emotions such as Sad and Frustrated compared to existing SOTA models?
• RQ5 How does the model perform across different modality combinations (e.g., text-audio, text-visual, full three-modality) on benchmark datasets?

Key findings

• GA2MIF achieves a weighted F1 score of 70.00% on the IEMOCAP dataset, significantly outperforming all baseline models, including MMGCN, under the three-modality setting.
• On the IEMOCAP dataset, the acoustic-textual setting yields 67.47% accuracy and 67.49% F1, which are 7.77 and 7.64 percentage points higher than the visual-textual setting.
• The model reduces the misclassification rate of ground-truth Sad emotion as Frustrated by 7.75% compared to MMGCN, dropping from 15.51% to 7.76%.
• The three-modality setting consistently outperforms all two-modality settings across both IEMOCAP and MELD datasets, confirming the benefit of full multimodal integration.
• GA2MIF demonstrates robustness and generalization across datasets, achieving state-of-the-art performance on both IEMOCAP and MELD with consistent improvements in accuracy and F1 scores.
• Ablation studies confirm that both MDGATs and MPCATs contribute significantly to performance, with the context window mechanism effectively reducing redundant connections without sacrificing model capacity.