[Paper Review] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Junnan Li, Dongxu Li+2 more|arXiv (Cornell University)|Jan 28, 2022

Multimodal Machine Learning ApplicationsComputer Science862 citations

TL;DR

BLIP presents a unified vision-language pre-training framework (MED) with a captioning-and-filtering data-bootstrapping (CapFilt) strategy to learn from noisy web image-text pairs, achieving state-of-the-art across understanding and generation tasks and strong zero-shot video-language transfer.

ABSTRACT

Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.

Motivation & Objective

• Aim to create a single model that excels at both vision-language understanding and generation tasks.
• Address suboptimality of noisy web text by bootstrapping captions via a captioner and a filter.
• Develop a flexible model architecture that supports multiple modalities and tasks.
• Leverage a data-augmentation pipeline (CapFilt) to improve pre-training data quality.
• Demonstrate strong transfer performance to video-language tasks in zero-shot settings.

Proposed method

• Introduce Multimodal Mixture of Encoder-Decoder (MED) enabling unimodal encoder, image-grounded text encoder, and image-grounded text decoder within a unified model.
• Pre-train with three objectives: Image-Text Contrastive (ITC), Image-Text Matching (ITM), and Language Modeling (LM) for caption generation.
• Share parameters between text encoder and decoder except SA layers to enable multi-task training efficiency.
• CapFilt bootstraps web data by: (i) captioner generating synthetic captions for web images, (ii) filter removing noisy captions via ITM/ITC signals, (iii) combining cleaned web data with human-annotated data for pre-training.
• Utilize nucleus sampling for synthetic captions to enhance diversity and model learning.
• Evaluate with extensive ablations on dataset size, backbone, and parameter sharing to optimize CapFilt effectiveness.

Experimental results

Research questions

• RQ1 Can a unified vision-language model (MED) perform well on both understanding and generation tasks?
• RQ2 Does bootstrapping noisy web data with synthetic captions and a filtering step improve downstream vision-language performance?
• RQ3 How do data diversity and parameter-sharing strategies affect pre-training efficiency and downstream results?
• RQ4 What is the impact of CapFilt on zero-shot and fine-tuned tasks, including retrieval, captioning, VQA, NLVR2, and VisDial?
• RQ5 How well does BLIP generalize to video-language tasks in zero-shot transfer?

Key findings

• BLIP achieves state-of-the-art results across vision-language tasks such as image-text retrieval, image captioning, VQA, visual reasoning, and visual dialog.
• Zero-shot transfer to video-language tasks (text-to-video retrieval and videoQA) reaches state-of-the-art performance.
• CapFilt consistently improves performance over using original noisy web texts, with larger gains when paired with larger backbones and more data.
• Diversity in synthetic captions (via nucleus sampling) yields gains over deterministic beam search.
• Sharing text encoder and decoder parameters except self-attention layers provides a favorable balance of performance and efficiency; decoupled captioner and filter generally perform better than fully shared settings.
• ViT-L backbones with CapFilt-bootstrapped data achieve strong results, approaching or matching methods trained on substantially larger datasets.