[Paper Review] VLP: A Survey on Vision-Language Pre-training

Feilong Chen, Duzhen Zhang+5 more|arXiv (Cornell University)|Feb 18, 2022

Multimodal Machine Learning Applications151 references4 citations

TL;DR

This survey provides a comprehensive overview of vision-language pre-training (VLP), covering feature extraction, model architectures, pre-training objectives, datasets, and downstream tasks. It systematically reviews state-of-the-art VLP models and identifies emerging frontiers such as prompt tuning, model compression, out-of-domain pretraining, and novel architectures like diffusion models and geometric deep learning.

ABSTRACT

In the past few years, the emergence of pre-training models has brought uni-modal fields such as computer vision (CV) and natural language processing (NLP) to a new era. Substantial works have shown they are beneficial for downstream uni-modal tasks and avoid training a new model from scratch. So can such pre-trained models be applied to multi-modal tasks? Researchers have explored this problem and made significant progress. This paper surveys recent advances and new frontiers in vision-language pre-training (VLP), including image-text and video-text pre-training. To give readers a better overall grasp of VLP, we first review its recent advances from five aspects: feature extraction, model architecture, pre-training objectives, pre-training datasets, and downstream tasks. Then, we summarize the specific VLP models in detail. Finally, we discuss the new frontiers in VLP. To the best of our knowledge, this is the first survey focused on VLP. We hope that this survey can shed light on future research in the VLP field.

Motivation & Objective

• To provide a systematic and comprehensive survey of vision-language pre-training (VLP) for researchers in computer vision and NLP.
• To identify and analyze key components of VLP, including feature extraction, model architecture, pre-training objectives, datasets, and downstream tasks.
• To summarize state-of-the-art VLP models and highlight their architectural and objective innovations.
• To explore emerging frontiers in VLP, such as prompt tuning, model compression, out-of-domain pretraining, and novel architectures.
• To serve as a foundational reference for future research by consolidating the current state of the field and identifying open challenges.

Proposed method

• Categorizes VLP models based on feature extraction techniques, including object detection-based region features and CLIP-style image encoders.
• Classifies model architectures into single-stream vs. dual-stream and encoder-only vs. encoder-decoder configurations.
• Reviews four main types of pre-training objectives: completion, matching, temporal, and specialized objectives like masked language modeling.
• Compiles and analyzes major pre-training datasets such as COCO, Conceptual Captions, and MS-COCO, emphasizing their scale and modality coverage.
• Examines downstream tasks including image captioning, visual question answering, and video-language understanding, detailing their evaluation goals.
• Explores emerging techniques such as prompt tuning, knowledge distillation, and model quantization for improving efficiency and generalization.

Experimental results

Research questions

• RQ1 What are the key components and design choices in modern vision-language pre-training models?
• RQ2 How do different pre-training objectives guide the learning of cross-modal alignment between vision and language?
• RQ3 What are the most influential VLP models, and what architectural or objective innovations do they introduce?
• RQ4 What are the major challenges and open research directions in VLP, particularly regarding efficiency, generalization, and cognitive integration?
• RQ5 How can emerging techniques like prompt tuning and knowledge infusion improve the performance and adaptability of VLP models?

Key findings

• This is the first comprehensive survey focused exclusively on vision-language pre-training, consolidating advances across feature extraction, model design, objectives, datasets, and downstream tasks.
• Object detection-based region features (e.g., Faster R-CNN) and CLIP-style contrastive learning are dominant in feature extraction, with the latter enabling zero-shot generalization.
• Encoder-only and encoder-decoder architectures are widely used, with dual-stream models often outperforming single-stream in tasks requiring modality-specific processing.
• Pre-training objectives such as masked language modeling, contrastive learning, and masked region modeling are critical for learning cross-modal alignment.
• Recent models like VATT and OPT demonstrate the feasibility of joint text, image, and audio pre-training with multi-level masking and generation capabilities.
• Emerging frontiers such as prompt tuning, knowledge infusion, and model compression show strong potential for improving efficiency, generalization, and scalability in VLP.