[Paper Review] M-BERT: Injecting Multimodal Information in the BERT Structure

Wasifur Rahman, K. M. Azharul Hasan+3 more|arXiv (Cornell University)|Aug 15, 2019

Multimodal Machine Learning Applications11 references31 citations

TL;DR

This paper proposes M-BERT, a method to inject multimodal information (text, vision, and audio) directly into the input space of BERT for multimodal sentiment analysis. By fusing modality-specific features before the BERT encoder, it achieves a new state of the art at 84.38% binary accuracy on the CMU-MOSI dataset, outperforming both vanilla BERT and prior multimodal models by 1.02% and 5.98% respectively.

ABSTRACT

Multimodal language analysis is an emerging research area in natural language processing that models language in a multimodal manner. It aims to understand language from the modalities of text, visual, and acoustic by modeling both intra-modal and cross-modal interactions. BERT (Bidirectional Encoder Representations from Transformers) provides strong contextual language representations after training on large-scale unlabeled corpora. Fine-tuning the vanilla BERT model has shown promising results in building state-of-the-art models for diverse NLP tasks like question answering and language inference. However, fine-tuning BERT in the presence of information from other modalities remains an open research problem. In this paper, we inject multimodal information within the input space of BERT network for modeling multimodal language. The proposed injection method allows BERT to reach a new state of the art of $84.38\%$ binary accuracy on CMU-MOSI dataset (multimodal sentiment analysis) with a gap of 5.98 percent to the previous state of the art and 1.02 percent to the text-only BERT.

Motivation & Objective

• To address the challenge of fine-tuning BERT for multimodal language understanding, where additional modalities like vision and audio are not effectively integrated.
• To improve performance on multimodal sentiment analysis tasks by enhancing BERT’s ability to model cross-modal interactions.
• To achieve state-of-the-art results on the CMU-MOSI benchmark by injecting multimodal signals directly into the input representation of BERT.
• To demonstrate that input-space injection of multimodal features is a simple yet effective alternative to architectural modifications in multimodal BERT models.

Proposed method

• The proposed method injects modality-specific features (text, visual, acoustic) into the input embedding layer of BERT before the transformer encoder.
• Multimodal features are concatenated with the original BERT input embeddings, preserving the model’s pre-trained attention mechanisms.
• The model uses early fusion, where features from different modalities are combined at the input level before being processed by the BERT encoder.
• The input representation is constructed by concatenating the BERT-pooled token embeddings with modality-specific features from vision and audio.
• The model is fine-tuned end-to-end on the CMU-MOSI dataset using standard BERT optimization and loss functions.
• The approach maintains the original BERT architecture, requiring no changes to attention mechanisms or transformer layers.

Experimental results

Research questions

• RQ1 Can multimodal information be effectively injected into the input space of BERT without architectural modifications?
• RQ2 How does input-level fusion of text, vision, and audio improve multimodal sentiment classification performance?
• RQ3 To what extent does the proposed method outperform vanilla BERT and prior multimodal BERT models on benchmark datasets?
• RQ4 Does early fusion of multimodal features lead to better cross-modal interaction learning compared to later fusion or modality-specific adaptation?
• RQ5 Can a simple input injection strategy achieve state-of-the-art results in multimodal sentiment analysis?

Key findings

• M-BERT achieves a new state of the art with 84.38% binary accuracy on the CMU-MOSI dataset for multimodal sentiment analysis.
• The model outperforms the vanilla BERT baseline by 1.02 percentage points, demonstrating the effectiveness of multimodal injection.
• The method surpasses the previous state of the art by 5.98 percentage points, indicating a significant performance gain.
• The results show that input-space injection of multimodal features is sufficient to achieve strong performance without modifying the BERT architecture.
• The approach maintains the efficiency and scalability of BERT while enabling robust multimodal understanding.
• The success of the method suggests that early fusion in the input space is a viable and effective strategy for multimodal NLP tasks.