[Paper Review] ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data

Di Qi, Lin Su+4 more|arXiv (Cornell University)|Jan 22, 2020

Multimodal Machine Learning Applications30 references155 citations

TL;DR

ImageBERT is a Transformer-based vision-language model pre-trained with a multi-stage pipeline on LAIT and other datasets using four tasks (MLM, MOC, MRFR, ITM) to achieve state-of-the-art image-text retrieval on MSCOCO and Flickr30k.

ABSTRACT

In this paper, we introduce a new vision-language pre-trained model -- ImageBERT -- for image-text joint embedding. Our model is a Transformer-based model, which takes different modalities as input and models the relationship between them. The model is pre-trained on four tasks simultaneously: Masked Language Modeling (MLM), Masked Object Classification (MOC), Masked Region Feature Regression (MRFR), and Image Text Matching (ITM). To further enhance the pre-training quality, we have collected a Large-scale weAk-supervised Image-Text (LAIT) dataset from Web. We first pre-train the model on this dataset, then conduct a second stage pre-training on Conceptual Captions and SBU Captions. Our experiments show that multi-stage pre-training strategy outperforms single-stage pre-training. We also fine-tune and evaluate our pre-trained ImageBERT model on image retrieval and text retrieval tasks, and achieve new state-of-the-art results on both MSCOCO and Flickr30k datasets.

Motivation & Objective

• Develop a Transformer-based model for joint image-text embedding.
• Leverage a large-scale weakly supervised image-text dataset (LAIT) plus public data through multi-stage pre-training.
• Design and evaluate four pre-training tasks to model language, vision, and cross-modal interactions.
• Fine-tune on image-text retrieval tasks and compare with prior state-of-the-art on MSCOCO and Flickr30k.

Proposed method

• Use a multi-modal Transformer that ingests both RoI-based visual tokens and textual tokens.
• Represent images via RoI features from Faster-RCNN with 5-D spatial embeddings, fused with word-piece text embeddings.
• Pre-train with four tasks: MLM (Masked Language Modeling), MOC (Masked Object Classification), MRFR (Masked Region Feature Regression), ITM (Image-Text Matching).
• Adopt a two-stage multi-stage pre-training: Stage-1 on LAIT, Stage-2 on Conceptual Captions and SBU Captions, then fine-tune on downstream retrieval tasks.
• Fine-tune with image-to-text and text-to-image retrieval objectives using various losses (Binary BCE, Cross-Entropy with multiple negatives, Triplet) to maximize ranking margins.

Experimental results

Research questions

• RQ1 Does multi-stage pre-training with LAIT improve cross-modal representations for retrieval tasks?
• RQ2 How do different pre-training data (LAIT, CC, SBU) affect zero-shot and fine-tuned performance on image-text retrieval?
• RQ3 What is the impact of architectural choices (RoI count, global features) and pre-training tasks (MRFR, MOC, MLM) on performance?
• RQ4 Can ImageBERT achieve state-of-the-art results on MSCOCO and Flickr30k for both image and sentence retrieval after fine-tuning?

Key findings

• Multi-stage pre-training outperforms single-stage pre-training in retrieval tasks.
• Pre-trained ImageBERT shows new state-of-the-art zero-shot results on MSCOCO (stage-2) though UNITER can exceed in some zero-shot settings.
• Fine-tuned ImageBERT achieves new state-of-the-art results on both Flickr30k and MSCOCO across 1k and 5k test sets.
• Including LAIT in pre-training and combining with CC and SBU improves performance (LAIT+CC+SBU beats LAIT alone).
• Using 100 RoIs and incorporating MRFR loss significantly boosts zero-shot and fine-tuned retrieval performance; binary+CE+triplet fine-tuning yields strong results.