[Paper Review] Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition

Yuntao Shou, Tao Meng+3 more|arXiv (Cornell University)|Dec 28, 2023

Emotion and Mood Recognition4 citations

TL;DR

This paper proposes AR-IIGCN, a novel multimodal emotion recognition framework that uses adversarial representation learning to reduce modality heterogeneity and graph contrastive learning to capture intra-modal, inter-modal, intra-class, and inter-class semantic relationships. The method achieves state-of-the-art performance on IEMOCAP and MELD datasets, with 70.4% weighted F1 on IEMOCAP using RoBERTa-Large, demonstrating superior feature representation and classification accuracy through joint modality alignment and contrastive learning.

ABSTRACT

With the release of increasing open-source emotion recognition datasets on social media platforms and the rapid development of computing resources, multimodal emotion recognition tasks (MER) have begun to receive widespread research attention. The MER task extracts and fuses complementary semantic information from different modalities, which can classify the speaker's emotions. However, the existing feature fusion methods have usually mapped the features of different modalities into the same feature space for information fusion, which can not eliminate the heterogeneity between different modalities. Therefore, it is challenging to make the subsequent emotion class boundary learning. To tackle the above problems, we have proposed a novel Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive for Multimodal Emotion Recognition (AR-IIGCN) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features through adversarial representation, which can achieve information interaction between modalities and eliminate heterogeneity among modalities. Thirdly, we introduce contrastive graph representation learning to capture intra-modal and inter-modal complementary semantic information and learn intra-class and inter-class boundary information of emotion categories. Specifically, we construct a graph structure for three modal features and perform contrastive representation learning on nodes with different emotions in the same modality and the same emotion in different modalities, which can improve the feature representation ability of nodes. Extensive experimental works show that the ARL-IIGCN method can significantly improve emotion recognition accuracy on IEMOCAP and MELD datasets.

Motivation & Objective

• To address the challenge of modality heterogeneity in multimodal emotion recognition, where features from text, audio, and video are mapped into a shared space, leading to misalignment and poor boundary learning.
• To improve feature representation by capturing both intra-modal and inter-modal complementary semantic information across modalities.
• To enhance class boundary learning by explicitly modeling intra-class compactness and inter-class separability through contrastive learning.
• To develop a generalizable framework applicable beyond emotion recognition, such as in multimodal humor detection.
• To validate the effectiveness of adversarial learning and graph contrastive learning components through ablation studies and cross-dataset evaluation.

Proposed method

• The model uses separate encoders—RoBERTa for text, DenseNet for video, and Bi-LSTM for audio—to extract modality-specific features and map them into distinct initial feature spaces.
• An adversarial representation learning module (TGAN) is introduced to align features across modalities by minimizing distribution discrepancies, reducing modality heterogeneity.
• A graph contrastive learning framework is constructed where nodes represent samples, with edges formed between samples of the same modality and same emotion (intra-modal) and between samples of different modalities but same emotion (inter-modal).
• Intra-modal contrastive learning maximizes similarity between samples of the same emotion within a modality, while inter-modal contrastive learning minimizes distance between cross-modal samples of the same emotion.
• The model employs a graph neural network to propagate and refine representations based on the constructed graph structure, enhancing node-level feature learning.
• A final multi-layer perceptron (MLP) performs end-to-end emotion classification on the refined multimodal representations.

Figure 1: Illustrative example of the effect of different feature fusion methods on sentiment classification. (a) The feature extraction process for text, video and audio modalities. (b) The model learns emotion class boundaries in a feature fusion manner to map a common feature space. (c) The model

Experimental results

Research questions

• RQ1 Can adversarial representation learning effectively reduce modality heterogeneity in multimodal emotion recognition?
• RQ2 How does joint intra-modal and inter-modal graph contrastive learning improve feature representation and class boundary learning?
• RQ3 What is the relative contribution of adversarial alignment, intra-modal contrastive learning, and inter-modal contrastive learning to overall performance?
• RQ4 Can the proposed AR-IIGCN framework be generalized to other multimodal tasks beyond emotion recognition?
• RQ5 Does the model achieve state-of-the-art performance on benchmark datasets such as IEMOCAP and MELD?

Key findings

• AR-IIGCN achieves a weighted F1 score of 70.4% on the IEMOCAP dataset when using RoBERTa-Large as the text encoder, outperforming the baseline and ablation variants.
• On the MELD dataset, the model achieves a weighted F1 score of 64.0% with RoBERTa-Large, demonstrating strong generalization across datasets.
• Ablation studies show that intra-modal and inter-modal contrastive learning (IMCL and ICCL) contribute most significantly to performance, with IMCL yielding the second-highest performance after the full model.
• The adversarial alignment (TGAN) component provides a modest improvement over the baseline, indicating its role in modality alignment but less impact than contrastive learning.
• The method generalizes well to other multimodal tasks, achieving a 7.57% improvement in humor detection accuracy on the UR-FUNNY dataset when applied to C-MFN.
• The full AR-IIGCN model consistently outperforms all ablation variants and baseline models across both IEMOCAP and MELD, confirming the synergy of its components.

Figure 2: The overall framework of the Adversarial Representation Learning with Intra-Modal and Inter-Modal Graph Contrastive Learning consists of a data preprocessing layer, a multimodal feature fusion layer, a graph contrastive representation learning layer, and an emotion classification layer.