[Paper Review] MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

Devamanyu Hazarika, Roger Zimmermann+1 more|arXiv (Cornell University)|May 7, 2020

Humor Studies and Applications62 citations

TL;DR

MISA learns modality-invariant and modality-specific representations for each modality and uses a Transformer-based fusion to improve multimodal sentiment analysis and humor detection, achieving state-of-the-art results on MOSI, MOSEI, and UR_FUNNY.

ABSTRACT

Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.

Motivation & Objective

• Motivate improved fusion in multimodal sentiment analysis by addressing modality gaps and heterogeneity.
• Propose a representation learning framework that factorizes modality information into invariant and specific subspaces.
• Demonstrate that invariant+specific representations improve prediction when fused with a simple attention-based mechanism.
• Show cross-domain effectiveness on sentiment analysis (MOSI/MOSEI) and humor detection (UR_FUNNY).

Proposed method

• Project each modality utterance to two subspaces: a shared modality-invariant space and a modality-specific private space.
• Use shared parameters for the invariant encoder across modalities and modality-specific encoders per modality.
• Represent each modality with six vectors (three invariant and three specific) via simple feed-forward encoders.
• Fuse the six vectors with a self-attention Transformer over the concatenated representations, producing a joint representation for prediction.
• Train with a composite loss: task loss plus similarity (CMD) loss to align invariant representations, difference (orthogonality) losses to enforce non-redundancy, and reconstruction loss to preserve modality details.
• Loss terms: L = L_task + α L_sim + β L_diff + γ L_recon, where L_sim uses CMD, L_diff enforces orthogonality, and L_recon reconstructs inputs.

Experimental results

Research questions

• RQ1 Can learning modality-invariant representations reduce modality gaps and improve fusion performance in multimodal sentiment analysis?
• RQ2 Does adding modality-specific representations provide complementary information that enhances prediction when fused with invariant features?
• RQ3 Is a simple Transformer-based fusion sufficient to leverage the invariant and specific representations for accurate predictions across sentiment analysis and humor detection?
• RQ4 How does MISA perform on standard MSA benchmarks (MOSI, MOSEI) and on humor detection (UR_FUNNY) compared to state-of-the-art baselines?

Key findings

• MISA achieves state-of-the-art performance on MOSI and MOSEI across regression and classification metrics compared to prior models.
• The GloVe and BERT language features both benefit from MISA, with BERT generally improving performance over GloVe.
• On UR_FUNNY, MISA with GloVe and with BERT features outperforms strong baselines, indicating effectiveness for multimodal humor detection.
• Ablation studies show that removing L_sim, L_diff, or L_recon degrades performance, confirming the utility of each component.
• Compared to the SOTA ICCN in MOSEI and C-MFN in MHD, MISA provides competitive gains across multiple metrics.
• The model operates at the utterance level and can surpass some more complex fusion approaches, highlighting the value of representation learning prior to fusion.