[Paper Review] MSAF: Multimodal Split Attention Fusion

Lang Su, Chuqing Hu+2 more|arXiv (Cornell University)|Dec 13, 2020

Emotion and Mood RecognitionPsychology54 references44 citations

TL;DR

MSAF introduces a lightweight, channel-wise split-attention fusion module that integrates multimodal features for CNN/RNN architectures, achieving strong results across emotion recognition, sentiment analysis, and action recognition while using few additional parameters.

ABSTRACT

Multimodal learning mimics the reasoning process of the human multi-sensory system, which is used to perceive the surrounding world. While making a prediction, the human brain tends to relate crucial cues from multiple sources of information. In this work, we propose a novel multimodal fusion module that learns to emphasize more contributive features across all modalities. Specifically, the proposed Multimodal Split Attention Fusion (MSAF) module splits each modality into channel-wise equal feature blocks and creates a joint representation that is used to generate soft attention for each channel across the feature blocks. Further, the MSAF module is designed to be compatible with features of various spatial dimensions and sequence lengths, suitable for both CNNs and RNNs. Thus, MSAF can be easily added to fuse features of any unimodal networks and utilize existing pretrained unimodal model weights. To demonstrate the effectiveness of our fusion module, we design three multimodal networks with MSAF for emotion recognition, sentiment analysis, and action recognition tasks. Our approach achieves competitive results in each task and outperforms other application-specific networks and multimodal fusion benchmarks.

Motivation & Objective

• Motivate efficient fusion of multimodal signals by emphasizing contributive features across modalities.
• Develop a lightweight module that can interoperate with CNN and RNN architectures using pretrained unimodal weights.
• Demonstrate MSAF effectiveness on emotion recognition, sentiment analysis, and action recognition datasets.
• Provide guidance on hyperparameters and deployment considerations for practical multimodal systems.

Proposed method

• Split each modality's feature channels into equal-sized blocks.
• Join modality blocks to form a multimodal descriptor and compute a shared channel descriptor via global pooling.
• Generate per-block attention across blocks using a shared transformation and softmax, with a regularizer lambda to control suppression.
• Reweight blocks within each modality and reassemble to produce enhanced modality features.
• Optionally apply BlockDropout to reduce overfitting and improve robustness.
• Extend MSAF to RNNs by segmenting sequences and applying block-wise attention within each segment.

Experimental results

Research questions

• RQ1 Can MSAF effectively fuse multimodal features with minimal parameter overhead across CNN and RNN architectures?
• RQ2 How does MSAF compare to state-of-the-art fusion methods on emotion recognition, sentiment analysis, and action recognition?
• RQ3 What hyperparameters (C, r, lambda, p, q) optimize performance and robustness across tasks?

Key findings

• MSAF achieves competitive to state-of-the-art results across three tasks with fewer parameters than many baselines.
• On emotion recognition (RAVDESS), MSAF outperforms MMTM while using fewer parameters (MSAF 25.94 M vs MMTM 31.97 M) and achieves 74.86% accuracy in the reported setup.
• On sentiment analysis (CMU-MOSEI), MSAF matches or exceeds prior multimodal methods with a simpler architecture.
• On action recognition (NTU RGB+D CS), MSAF with intermediate and late fusion yields state-of-the-art performance (92.24% CS).
• BlockDropout and lambda help mitigate over-suppression and overfitting, particularly on smaller datasets.