[Paper Review] Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

Zhun Liu, Ying Shen+4 more|arXiv (Cornell University)|May 31, 2018

Sentiment Analysis and Opinion MiningComputer Science25 references85 citations

TL;DR

The paper introduces Low-rank Multimodal Fusion (LMF), a method that uses modality-specific low-rank factors to fuse multiple modalities efficiently, achieving competitive results with linear scalability in the number of modalities and significantly reduced parameters and compute compared to tensor-based fusion like TFN.

ABSTRACT

Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exploited the expressiveness of tensors for multimodal representation. However, these methods often suffer from exponential increase in dimensions and in computational complexity introduced by transformation of input into tensor. In this paper, we propose the Low-rank Multimodal Fusion method, which performs multimodal fusion using low-rank tensors to improve efficiency. We evaluate our model on three different tasks: multimodal sentiment analysis, speaker trait analysis, and emotion recognition. Our model achieves competitive results on all these tasks while drastically reducing computational complexity. Additional experiments also show that our model can perform robustly for a wide range of low-rank settings, and is indeed much more efficient in both training and inference compared to other methods that utilize tensor representations.

Motivation & Objective

• Motivate and address the scalability problem in multimodal fusion when using full tensor representations.
• Propose a low-rank, modality-specific factorization to enable linear scaling with the number of modalities.
• Show that LMF achieves competitive performance on sentiment, speaker trait, and emotion recognition while reducing parameters and computation.

Proposed method

• Formulate multimodal fusion as a multilinear function and identify the exponential cost of full tensor fusion.
• Decompose the weight tensor into modality-specific low-rank factors and derive an efficient computation that avoids forming the full input tensor.
• Derive an efficient fusion equation that computes h directly from unimodal representations using r rank-specific factors and elementwise products, reducing complexity to O(d_y * r * sum(d_m)).
• Provide a slightly different implementation form by concatenating factors into M order-3 tensors and performing a Lambda-based combination for practical computation.

Experimental results

Research questions

• RQ1 How does low-rank, modality-specific fusion compare to full tensor fusion (TFN) in performance on multimodal tasks?
• RQ2 Can LMF scale linearly with the number of modalities while maintaining competitive accuracy and regression metrics?
• RQ3 What is the impact of different rank settings on performance and stability?
• RQ4 How does LMF compare to state-of-the-art multimodal fusion methods in terms of parameters and speed?

Key findings

• LMF significantly outperforms TFN on the evaluated tasks across datasets, highlighting the advantage of low-rank fusion.
• LMF achieves competitive to state-of-the-art results for sentiment (MOSI), emotion (IEMOCAP), and speaker trait (POM) recognition.
• Theoretical and empirical analyses show LMF scales linearly with the number of modalities and reduces parameter count (about 11x fewer parameters than TFN for a three-modality setting).
• LMF delivers faster training and testing speeds compared to TFN (training and testing IPS in the reported setup).
• Rank settings: very low ranks suffice for competent performance, with higher ranks causing instability in some cases.