[Paper Review] Learning Factorized Multimodal Representations

Yao-Hung Hubert Tsai, Paul Pu Liang+3 more|arXiv (Cornell University)|Jun 16, 2018

Sentiment Analysis and Opinion Mining74 references197 citations

TL;DR

The paper introduces the Multimodal Factorization Model (MFM) that factorizes representations into multimodal discriminative factors and modality-specific generative factors, optimizing a joint generative-discriminative objective to improve prediction and enable missing-modality reconstruction.

ABSTRACT

Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing. In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction. Modality-specific generative factors are unique for each modality and contain the information required for generating data. Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets. Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance. Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

Motivation & Objective

• Address the challenge of learning rich intra- and cross-modal representations for prediction.
• Develop a model that remains robust when modalities are missing or noisy at test time.
• Factorize representations into shared multimodal discriminative factors and modality-specific generative factors.
• Enable flexible generation and reconstruction by conditioning on independent latent factors.
• Provide interpretability of learned factorized representations.

Proposed method

• Propose the Multimodal Factorization Model (MFM) with latent variables Z = [Z_y, Z_a1,...,Z_aM] generating discriminative factors F_y and modality-specific generative factors F_a{1:M}.
• Factorization yields P(X_hat_{1:M}, Y_hat) = ∫ P(X_hat_{1:M}, Y_hat | F) P(F | Z) P(Z) dF dZ with F_y and F_a accordingly.
• Use a joint-distribution Wasserstein distance objective to align P(X_{1:M}, Y) and P(X_hat_{1:M}, Y_hat), approximated via a generalized mean-field inference Q(Z | X_{1:M}, Y}).
• Adopt a surrogate inference network to reconstruct missing modalities given observed ones and to predict labels.
• Employ encoder Q(Z_y | X_{1:M}) and Q(Z_a_i | X_i), with decoders G_y, G_a_i, D, and F_• for reconstruction and prediction.
• Train with a hybrid objective combining modality reconstruction loss (generative) and label prediction loss (discriminative).
• Demonstrate model-agnostic applicability by integrating with various multimodal encoders (e.g., MFN, EF-LSTM, TFN).

Experimental results

Research questions

• RQ1 Can factorizing multimodal representations into a shared discriminative factor and modality-specific generative factors improve discriminative performance across datasets?
• RQ2 Does the factorization enable robust reconstruction and prediction when some modalities are missing at test time?
• RQ3 To what extent do the latent factors provide interpretable insights into multimodal interactions and contributions of each modality?
• RQ4 Is the approach compatible with different multimodal encoders and time-series modalities?
• RQ5 How do ablations of factorization and generative/discriminative components affect performance?

Key findings

• MFM achieves state-of-the-art or competitive results across six multimodal datasets (time-series and synthetic image data).
• Factorizing into multimodal discriminative and modality-specific generative factors improves both reconstruction and prediction compared to baselines.
• The model supports missing-modality reconstruction with only modest loss in predictive performance, outperforming purely generative or purely discriminative baselines in reconstruction and sentiment prediction.
• Hybrid objective combining generative reconstruction and discriminative prediction yields better results than purely discriminative or purely generative variants.
• Ablation studies indicate benefits from each design component, with modality-specific generative factors and factorized representations providing the strongest gains.
• Interpretation methods (information-theoretic and gradient-based) reveal language as a major contributor to sentiment prediction in CMU-MOSI and demonstrate factor influence over generated outputs.