[Paper Review] Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion

Sijie Mai, Haifeng Hu+1 more|arXiv (Cornell University)|Nov 18, 2019

Human Pose and Action Recognition42 references19 citations

TL;DR

This paper proposes an adversarial representation graph fusion (ARGF) framework that learns a modality-invariant embedding space through adversarial training to reduce modality gap, followed by a hierarchical graph fusion network (GFN) that explicitly models unimodal, bimodal, and trimodal interactions. The method achieves state-of-the-art performance on IEMOCAP and CMU-MOSI datasets, with improvements of over 2% and 1% respectively over prior methods.

ABSTRACT

Learning joint embedding space for various modalities is of vital importance for multimodal fusion. Mainstream modality fusion approaches fail to achieve this goal, leaving a modality gap which heavily affects cross-modal fusion. In this paper, we propose a novel adversarial encoder-decoder-classifier framework to learn a modality-invariant embedding space. Since the distributions of various modalities vary in nature, to reduce the modality gap, we translate the distributions of source modalities into that of target modality via their respective encoders using adversarial training. Furthermore, we exert additional constraints on embedding space by introducing reconstruction loss and classification loss. Then we fuse the encoded representations using hierarchical graph neural network which explicitly explores unimodal, bimodal and trimodal interactions in multi-stage. Our method achieves state-of-the-art performance on multiple datasets. Visualization of the learned embeddings suggests that the joint embedding space learned by our method is discriminative. code is available at: \url{https://github.com/TmacMai/ARGF_multimodal_fusion}

Motivation & Objective

• To address the modality gap caused by heterogeneous distributions across modalities in multimodal fusion.
• To learn a joint, modality-invariant embedding space that enhances cross-modal compatibility and representation discrimination.
• To explicitly model hierarchical cross-modal interactions—unimodal, bimodal, and trimodal—through a structured graph fusion mechanism.
• To improve multimodal fusion performance through adversarial distribution matching, reconstruction, and classification losses.

Proposed method

• Uses an adversarial encoder-decoder-classifier framework to align source modality distributions with the target modality’s distribution via discriminative training.
• Applies adversarial training where encoders of source modalities are optimized to fool a discriminator into classifying their representations as real (target modality) samples.
• Introduces modality-specific decoders to reconstruct raw input features, preserving unimodal information during embedding learning.
• Implements a classifier head to ensure the learned embedding space is discriminative for downstream tasks.
• Employs a hierarchical graph fusion network (GFN) with three layers: unimodal, bimodal, and trimodal vertices, enabling structured fusion of cross-modal dynamics.
• Assigns learnable weights to graph edges, allowing dynamic importance assignment to different interaction types and enabling interpretable fusion.

Experimental results

Research questions

• RQ1 Can adversarial training effectively reduce the modality gap by aligning heterogeneous modality distributions?
• RQ2 How does joint embedding learning with reconstruction and classification losses improve representation quality in multimodal fusion?
• RQ3 Can a hierarchical graph fusion network explicitly model and improve unimodal, bimodal, and trimodal interactions?
• RQ4 Does the proposed framework outperform existing fusion strategies in multimodal sentiment analysis tasks?
• RQ5 To what extent is the learned embedding space discriminative and interpretable, as shown by visualization?

Key findings

• ARGF achieves state-of-the-art performance on IEMOCAP and CMU-MOSI datasets, outperforming prior methods by over 2% and 1% respectively in accuracy.
• The visualization of the learned embedding space shows clear, well-separated clusters for positive and negative sentiments, indicating high discriminative power.
• The embedding space without the classifier fails to distinguish sentiment classes effectively, demonstrating the necessity of the classification loss for discriminative learning.
• Graph fusion visualization reveals that language modality is most predictive in unimodal fusion, but other modalities become dominant when language is ambiguous.
• Bimodal fusion shows high weights for audio-language and visual-language interactions, indicating their strong contribution to sentiment prediction.
• Trimodal fusion is dominated by fusing two bimodal vertices, highlighting the importance of modeling bimodal interaction pairs, which DFG fails to capture.