[Paper Review] SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

Zirui Wang, Jiahui Yu+4 more|arXiv (Cornell University)|Aug 24, 2021

Multimodal Machine Learning Applications54 references342 citations

TL;DR

SimVLM pretrains vision-language representations end-to-end with a single Prefix Language Modeling objective on weakly aligned image-text data, achieving state-of-the-art results across VL benchmarks and strong zero-shot transfer.

ABSTRACT

With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the scalability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-specific objectives. In this work, we relax these constraints and present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM). Unlike prior work, SimVLM reduces the training complexity by exploiting large-scale weak supervision, and is trained end-to-end with a single prefix language modeling objective. Without utilizing extra data or task-specific customization, the resulting model significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA (+3.74% vqa-score), NLVR2 (+1.17% accuracy), SNLI-VE (+1.37% accuracy) and image captioning tasks (+10.1% average CIDEr score). Furthermore, we demonstrate that SimVLM acquires strong generalization and transfer ability, enabling zero-shot behavior including open-ended visual question answering and cross-modality transfer.

Motivation & Objective

• Motivate a simple, scalable vision-language pretraining framework that reduces reliance on expensive annotations and complex objectives.
• Demonstrate that end-to-end Prefix Language Modeling on raw images and text can match or exceed MLM-based VLP methods.
• Show strong zero-shot generalization and cross-modality transfer with large-scale weak supervision.

Proposed method

• Use a Transformer backbone to process raw image patches and text tokens without object detectors.
• Adopt Prefix Language Modeling, enabling bidirectional prefix encoding and autoregressive text generation on the remaining tokens.
• Pretrain from scratch on large-scale weakly aligned image-text and text-only data with a single LM loss.
• Integrate image patches via a Conv stage before patch embedding and apply 2D relative attention for image tokens.
• Fine-tune on standard VL benchmarks in a single-stage pretraining–finetuning pipeline.

Experimental results

Research questions

• RQ1 Can a minimal, generative vision-language pretraining framework trained only with a language modeling objective achieve SOTA on VL benchmarks?
• RQ2 Does PrefixLM enable effective zero-shot and cross-modality transfer without task-specific losses or object detectors?
• RQ3 How does using weakly labeled image-text data (and text-only data) compare to detection-based pretraining for VL tasks?
• RQ4 What is the impact of architectural choices (image patching, Conv stage, positional encoding) on VL performance?
• RQ5 Can the model exhibit open-ended VQA and cross-modality transfer in zero-shot settings?

Key findings

• SimVLM achieves state-of-the-art results on six VL benchmarks without extra data or task-specific losses.
• On VQA, SimVLM_base, Large, and Huge surpass prior methods, with Huge reaching VQA scores above 80%.
• On NLVR2 and SNLI-VE, SimVLM attains new SOTA/near-SOTA accuracies across model sizes.
• Image captioning and NoCaps/Multi30k show large gains, including an average CIDEr improvement of around 10 points.
• Zero-shot cross-modality transfer and open-ended VQA capabilities emerge with scaling and weak supervision.
• Cross-modality transfer (text-only fine-tuning then evaluation on VL tasks) achieves competitive results with supervised baselines.