[Paper Review] Factorized Multimodal Transformer for Multimodal Sequential Learning

Amir Zadeh, Chengfeng Mao+5 more|arXiv (Cornell University)|Nov 22, 2019

Speech and dialogue systemsComputer Science47 references37 citations

TL;DR

FMT introduces a Factorized Multimodal Self-attention mechanism to model intramodal and intermodal dynamics in asynchronous multimodal sequences, achieving state-of-the-art results on CMU-MOSI, IEMOCAP, and POM datasets.

ABSTRACT

The complex world around us is inherently multimodal and sequential (continuous). Information is scattered across different modalities and requires multiple continuous sensors to be captured. As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area. Arguably, modeling arbitrarily distributed spatio-temporal dynamics within and across modalities is the biggest challenge in this research area. In this paper, we present a new transformer model, called the Factorized Multimodal Transformer (FMT) for multimodal sequential learning. FMT inherently models the intramodal and intermodal (involving two or more modalities) dynamics within its multimodal input in a factorized manner. The proposed factorization allows for increasing the number of self-attentions to better model the multimodal phenomena at hand; without encountering difficulties during training (e.g. overfitting) even on relatively low-resource setups. All the attention mechanisms within FMT have a full time-domain receptive field which allows them to asynchronously capture long-range multimodal dynamics. In our experiments we focus on datasets that contain the three commonly studied modalities of language, vision and acoustic. We perform a wide range of experiments, spanning across 3 well-studied datasets and 21 distinct labels. FMT shows superior performance over previously proposed models, setting new state of the art in the studied datasets.

Motivation & Objective

• Motivate and address the challenge of modeling asynchronous spatio-temporal interactions across language, vision, and audio modalities.
• Propose a single-transformer architecture (FMT) with Factorized Multimodal Self-attention (FMS) to capture unimodal, bimodal, and trimodal interactions.
• Enable scalable modeling of long-range multimodal dynamics with full time-domain receptive fields while avoiding overfitting on limited resources.

Proposed method

• Embed each modality with unimodal embeddings and add positional information.
• Use Multimodal Transformer Layers (MTL) containing multiple Factorized Multimodal Self-attentions (FMS) to capture factorized intra- and intermodal dynamics.
• Within each FMS, compute seven attentions corresponding to L, V, A, LV, LA, VA, and LVA factors with full sequence length.
• Apply a 1D convolutional summarization networks (S1 and S2) to reduce the high-dimensional FMS outputs to a manageable representation.
• Feed the final MTL output to a GRU-based predictor for timestamped supervision and final sequence labeling.
• Compare FMT to strong baselines on CMU-MOSI, IEMOCAP, and POM using standard multimodal metrics.

Experimental results

Research questions

• RQ1 Can a factorized attention mechanism within a single transformer effectively model unimodal, bimodal, and trimodal interactions in asynchronous multimodal sequences?
• RQ2 Does a compact, fully time-domain-attentive architecture outperform prior multimodal sequential models on sentiment, emotion, and personality traits recognition?
• RQ3 How does varying the number of FMS units within an MTL affect performance and training efficiency?
• RQ4 What is the impact of removing unimodal/bimodal/trimodal factors on overall performance?

Key findings

• FMT achieves superior performance over baselines on CMU-MOSI for multimodal sentiment analysis (Table 1).
• FMT surpasses baselines on IEMOCAP for discrete emotions except Happy (Table 2).
• FMT outperforms baselines on POM for 16 traits (Table 3).
• Ablation studies show all factor types (UNI, BI, TRI) and summarization components are needed for best performance (Table 4).
• Increasing the number of FMS units up to 6 within an MTL yields peak performance in their experiments (Table 5).
• FMT uses fewer total attentions than MulT yet achieves better performance on the same tasks.