[Paper Review] VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

Hangbo Bao, Wenhui Wang+6 more|arXiv (Cornell University)|Nov 3, 2021

Multimodal Machine Learning ApplicationsComputer Science44 references288 citations

TL;DR

VLMo introduces a unified vision-language pre-training model with a Mixture-of-Modality-Experts (MoME) Transformer that can function as both a dual encoder for efficient retrieval and a fusion encoder for classification, enhanced by stagewise pre-training on image-only and text-only data.

ABSTRACT

We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network. Specifically, we introduce Mixture-of-Modality-Experts (MoME) Transformer, where each block contains a pool of modality-specific experts and a shared self-attention layer. Because of the modeling flexibility of MoME, pretrained VLMo can be fine-tuned as a fusion encoder for vision-language classification tasks, or used as a dual encoder for efficient image-text retrieval. Moreover, we propose a stagewise pre-training strategy, which effectively leverages large-scale image-only and text-only data besides image-text pairs. Experimental results show that VLMo achieves state-of-the-art results on various vision-language tasks, including VQA, NLVR2 and image-text retrieval. The code and pretrained models are available at https://aka.ms/vlmo.

Motivation & Objective

• Motivate a unified VL pre-training that supports both retrieval (dual encoder) and classification (fusion encoder) tasks.
• Introduce MoME Transformer to encode image, text, and image-text pairs within a shared backbone.
• Propose stagewise pre-training leveraging large-scale image-only and text-only data to improve VL representations.
• Pre-train and fine-tune VLMo on standard VL benchmarks to demonstrate state-of-the-art results.

Proposed method

• Introduce Mixture-of-Modality-Experts (MoME) Transformer with modality-specific experts (vision, language, vision-language) and a shared self-attention layer.
• Enforce modality-specific processing via experts while sharing attention to align visual and linguistic content.
• Pre-train with three tasks: image-text contrastive learning (ITC), image-text matching (ITM), and masked language modeling (MLM).
• Stagewise pre-training: first image-only pre-training for V-FFN and self-attention, then text-only pre-training for L-FFN, followed by VL pre-training.
• Fine-tune as dual encoder for retrieval (images and texts encoded separately) or as fusion encoder for VL classification tasks.
• Scale up data (up to 1B image-text pairs) and batch size to improve performance.

Experimental results

Research questions

• RQ1 Can a single MoME Transformer backbone effectively support both dual-encoder retrieval and fusion-encoder classification in vision-language tasks?
• RQ2 Does stagewise pre-training on image-only and text-only data improve vision-language representations compared to image-text-only pre-training?
• RQ3 What gains arise from incorporating a vision-language expert in top MoME layers for cross-modal interaction?

Key findings

• VLMo achieves state-of-the-art results on VL classification benchmarks (VQA, NLVR2) and competitive/leading results on image-text retrieval (COCO, Flickr30K) when used as a dual encoder.
• As a fusion encoder, VLMo delivers strong VL reasoning performance, outperforming several prior fusion-based models.
• Stagewise pre-training with image-only and text-only data substantially improves downstream performance.
• VLMo-Large++ trained on ~1B image-text pairs with larger batch sizes sets new records on several benchmarks.
• When used as a dual encoder, VLMo offers faster retrieval due to linear-time encoding versus quadratic fusion-based approaches.