[Paper Review] Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao+9 more|arXiv (Cornell University)|Aug 22, 2022

Multimodal Machine Learning ApplicationsComputer Science150 citations

TL;DR

BEiT-3 is a general-purpose multimodal foundation model that uses masked data modeling across images, text, and image-text pairs with Multiway Transformers, achieving state-of-the-art transfer to both vision and vision-language tasks.

ABSTRACT

A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked "language" modeling on images (Imglish), texts (English), and image-text pairs ("parallel sentences") in a unified manner. Experimental results show that BEiT-3 obtains state-of-the-art performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).

Motivation & Objective

• Motivate the unification of vision and language pretraining under a single foundation model
• Propose a general-purpose backbone (Multiway Transformers) that supports modality-specific encoding and cross-modal fusion
• Advocate a single pretraining objective: masked data modeling applied to images, text, and image-text pairs
• Show that scaling model and data size improves transfer to diverse downstream tasks
• Demonstrate public-resource-based pretraining achieving competitive to SOTA results across tasks

Proposed method

• Adopt Multiway Transformers with modality-specific experts and shared self-attention to enable deep fusion and modality-specific encoding
• Define a unified pretraining task: mask-then-predict (masked data modeling) over monomodal and multimodal data, treating images as a foreign language (Imglish)
• Tokenize text with SentencePiece and images with BEiT v2 visual tokens as targets for reconstruction
• Mask 15% of text tokens (monomodal) and 50% of image-text pair text tokens; mask 40% of image patches with block-wise masking
• Pretrain BEiT-3 with a 40-layer architecture (~1.9B parameters) on public data sources; use a batch setup of 2048 images, 2048 texts, 2048 image-text pairs per step
• Enable downstream transfer in various modes (vision encoder, dual encoder for retrieval, fusion encoder for multimodal tasks)

Experimental results

Research questions

• RQ1 Can a single, unified architecture (Multiway Transformers) efficiently support both vision and vision-language tasks?
• RQ2 Does a single masked data modeling objective on images, texts, and image-text pairs suffice to learn transferable multimodal representations?
• RQ3 How does scaling (model size and data) impact performance across a broad set of vision and vision-language benchmarks?

Key findings

• BEiT-3 achieves state-of-the-art transfer on diverse tasks including COCO object detection (AP), COCO instance segmentation (AP), ADE20K semantic segmentation (mIoU), ImageNet classification (Top-1), NLVR2 visual reasoning, VQAv2 visual question answering, COCO image captioning (CIDEr), and cross-modal retrieval (Flickr30K, COCO)
• On vision tasks, BEiT-3 matches or exceeds prior SOTA while using only public data resources for pretraining and finetuning
• On vision-language tasks, BEiT-3 surpasses previous SOTA across VQA, NLVR2, captioning, and retrieval benchmarks, demonstrating strong cross-modal alignment via masking-based pretraining
• Zero-shot and finetuned retrieval results show BEiT-3 competitive or superior to prior models, with notable gains in COCO and Flickr30K retrieval
• Semantic segmentation and image classification results show BEiT-3 delivering top-tier accuracy with a single unified model