[Paper Review] InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining

Junyang Lin, Yang An+4 more|arXiv (Cornell University)|Mar 30, 2020

Multimodal Machine Learning ApplicationsComputer Science60 references56 citations

TL;DR

InterBERT introduces a single-stream interaction mechanism with a two-stream extraction module for vision-language pretraining, along with MGM and ITM-hn pretraining tasks. It outperforms baselines on image retrieval and VCR, while preserving strong single-modal performance and is deployed in Taobao.

ABSTRACT

Multi-modal pretraining for learning high-level multi-modal representation is a further step towards deep learning and artificial intelligence. In this work, we propose a novel model, namely InterBERT (BERT for Interaction), which is the first model of our series of multimodal pretraining methods M6 (MultiModality-to-MultiModality Multitask Mega-transformer). The model owns strong capability of modeling interaction between the information flows of different modalities. The single-stream interaction module is capable of effectively processing information of multiple modalilties, and the two-stream module on top preserves the independence of each modality to avoid performance downgrade in single-modal tasks. We pretrain the model with three pretraining tasks, including masked segment modeling (MSM), masked region modeling (MRM) and image-text matching (ITM); and finetune the model on a series of vision-and-language downstream tasks. Experimental results demonstrate that InterBERT outperforms a series of strong baselines, including the most recent multi-modal pretraining methods, and the analysis shows that MSM and MRM are effective for pretraining and our method can achieve performances comparable to BERT in single-modal tasks. Besides, we propose a large-scale dataset for multi-modal pretraining in Chinese, and we develop the Chinese InterBERT which is the first Chinese multi-modal pretrained model. We pretrain the Chinese InterBERT on our proposed dataset of 3.1M image-text pairs from the mobile Taobao, the largest Chinese e-commerce platform. We finetune the model for text-based image retrieval, and recently we deployed the model online for topic-based recommendation.

Motivation & Objective

• Motivate robust multi-modal representation learning beyond simple MLM/MOM by enabling strong cross-modal interaction.
• Design InterBERT with a single-stream interaction module and a two-stream extraction module to preserve modality independence.
• Introduce pretraining tasks (masked group modeling and hard-negative image-text matching) to enhance cross-modal understanding.
• Evaluate on downstream vision-language tasks (caption-based retrieval and VCR) and analyze single-modal transferability and initialization effects.
• Demonstrate deployment potential via online Taobao deployment and A/B testing.

Proposed method

• Use a single-stream all-attention interaction module to fuse image and text embeddings.
• Implement a two-stream extraction module to produce modality-specific representations for downstream use.
• Pretrain with Masked Group Modeling (MSM for text, MRM for image) and Image-Text Matching with Hard Negatives (ITM-hn).
• MSM masks contiguous text segments; MRM masks image regions with high IoU to anchors.
• ITM-hn uses hard negatives retrieved via TF-IDF to create challenging image-text pairs.
• Fine-tune on downstream tasks such as caption-based image retrieval, zero-shot retrieval, and Visual Commonsense Reasoning (VCR).

Experimental results

Research questions

• RQ1 Can a multi-modal pretraining model benefit from a unified all-attention interaction while preserving modality independence?
• RQ2 Do MGM and ITM-hn pretraining tasks improve cross-modal understanding and downstream performance?
• RQ3 How well does InterBERT transfer to single-modal NLP tasks compared to BERT?
• RQ4 What is the impact of BERT initialization on multi-modal pretraining performance?
• RQ5 How does InterBERT perform on standard vision-language benchmarks (IR, zero-shot IR, VCR) relative to VilBERT/VL-BERT?

Key findings

• InterBERT outperforms strong baselines on image retrieval and VCR, including notable gains in zero-shot image retrieval.
• On Flickr30K-based image retrieval, InterBERT achieves 61.9% R@1, 87.1% R@5, and 92.7% R@10 (IR).
• In zero-shot image retrieval, InterBERT reaches 49.2% R@1, 77.6% R@5, and 86.0% R@10.
• For VCR, InterBERT achieves 73.1% Q→A, 74.8% QA→R, and 54.9% Q→AR, outperforming R2C and VilBERT baselines.
• InterBERT without pretraining underperforms compared to with pretraining, showing the effectiveness of multi-modal pretraining.
• GLUE-style results show InterBERT rivals BERT-base on NLP tasks, and preserve single-modal capabilities comparable to BERT-base.