[Paper Review] Fusion with Hierarchical Graphs for Mulitmodal Emotion Recognition

Shuyun Tang, Zhaojie Luo+3 more|arXiv (Cornell University)|Sep 15, 2021

Emotion and Mood Recognition38 references4 citations

TL;DR

This paper proposes Hierarchical Fusion Graph Convolutional Network (HFGCN), a novel graph-based model that captures intermodality interactions and modality-emotion relations through a two-stage graph construction with attention-based edge weights and relational graph transformation. HFGCN achieves state-of-the-art performance on IEMOCAP and MELD datasets by jointly predicting emotion labels and valence-arousal (VA) degrees via multitask learning, enhancing both accuracy and interpretability.

ABSTRACT

Automatic emotion recognition (AER) based on enriched multimodal inputs, including text, speech, and visual clues, is crucial in the development of emotionally intelligent machines. Although complex modality relationships have been proven effective for AER, they are still largely underexplored because previous works predominantly relied on various fusion mechanisms with simply concatenated features to learn multimodal representations for emotion classification. This paper proposes a novel hierarchical fusion graph convolutional network (HFGCN) model that learns more informative multimodal representations by considering the modality dependencies during the feature fusion procedure. Specifically, the proposed model fuses multimodality inputs using a two-stage graph construction approach and encodes the modality dependencies into the conversation representation. We verified the interpretable capabilities of the proposed method by projecting the emotional states to a 2D valence-arousal (VA) subspace. Extensive experiments showed the effectiveness of our proposed model for more accurate AER, which yielded state-of-the-art results on two public datasets, IEMOCAP and MELD.

Motivation & Objective

• To address the limitations of traditional fusion methods that rely on simple concatenation and fail to model complex intermodality interactions in multimodal emotion recognition.
• To explore how modality-emotion relationships—where the contribution of each modality varies by emotion type—can improve classification performance.
• To enhance model interpretability by projecting emotional states into a 2D valence-arousal (VA) subspace.
• To develop a hierarchical graph fusion mechanism that captures both utterance-level and conversation-level modality interactions.

Proposed method

• The model uses a two-stage graph construction: first, modality-specific features (text, audio, video) are connected via attention-based edges to model intermodality interactions at the utterance level.
• Second, utterance representations are connected through another set of attention-based edges to model conversation-level interactions, enabling hierarchical fusion across time.
• Edge relations are explicitly designed to encode modality-emotion dependencies, allowing the model to learn how different modalities contribute to different emotional states.
• A multitask learning loss is introduced to jointly predict discrete emotion labels and continuous valence-arousal (VA) degrees, improving representation learning and interpretability.
• Relational graph transformation is applied to capture the dynamic relationships between modalities and emotions across the conversation.
• The model leverages graph convolutional networks (GCNs) with learnable attention weights to dynamically assign importance to modality interactions and relational edges.

Experimental results

Research questions

• RQ1 How can hierarchical graph fusion improve multimodal representation learning for emotion recognition compared to traditional early or late fusion methods?
• RQ2 To what extent do modality-emotion relationships—where the importance of each modality varies by emotion type—enhance classification performance?
• RQ3 Can joint prediction of discrete emotion labels and continuous valence-arousal (VA) degrees improve model accuracy and interpretability?
• RQ4 How do attention-based edge weights and relational graph design contribute to capturing intermodality and modality-emotion dependencies?
• RQ5 What is the relative contribution of utterance-level versus conversation-level graph structures in modeling multimodal emotion dynamics?

Key findings

• HFGCN achieves state-of-the-art performance on the IEMOCAP dataset with an average F1 score of 74.90, significantly outperforming baseline methods.
• The ablation study confirms that removing both stages of graph construction reduces F1 by over 12 percentage points, demonstrating the necessity of hierarchical fusion.
• Removing attention-based edge weights leads to a 2.7% drop in F1, while removing edge relations causes a 3.9% drop, indicating that relational design is more critical than attention weights alone.
• The model's visualization in the 2D valence-arousal (VA) space reveals that it captures general emotional distributions, though it tends to over-generalize along the arousal axis.
• Misclassifications are most common for short utterances like 'Okay' or 'I just can’t', which lie in transitional zones between neutral, angry, and excited emotions.
• The joint prediction of emotion labels and VA degrees enhances model interpretability, as shown by the alignment of predicted emotional states with the theoretical VA framework (Barrett, 1998).