[Paper Review] GLIPv2: Unifying Localization and Vision-Language Understanding

Haotian Zhang, Pengchuan Zhang+8 more|arXiv (Cornell University)|Jun 12, 2022

Multimodal Machine Learning ApplicationsComputer Science126 citations

TL;DR

GLIPv2 is a pre-trained grounded vision-language model that unifies localization (detection/segmentation) and VL understanding (grounded VQA, grounding, and captioning) for language-guided visual tasks. It demonstrates that localization and VL understanding mutually benefit each other across diverse datasets.

ABSTRACT

We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning). GLIPv2 elegantly unifies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as a VL reformulation of the detection task, region-word contrastive learning as a novel region-word level contrastive learning task, and the masked language modeling. This unification not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks. Experimental results show that a single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks. The model also shows (1) strong zero-shot and few-shot adaption performance on open-vocabulary object detection tasks and (2) superior grounding capability on VL understanding tasks. Code will be released at https://github.com/microsoft/GLIP.

Motivation & Objective

• Motivate a single model to perform both localization and vision-language understanding tasks.
• Enable language-guided detection, segmentation, grounding, VQA, and captioning.
• Investigate pre-training losses and data scales for joint localization and VL understanding.
• Provide detailed training, ablation, and inference-speed analyses across multiple datasets.

Proposed method

• Pre-train GLIPv2 variants on detection, grounding, and caption data with three loss components: intra-image region-word contrastive, inter-image word-region contrastive, and MLM.
• Use a unified architecture with an Hourglass-based segmentation head and a VL fusion module to connect visual and linguistic features.
• Fine-tune downstream tasks such as COCO/ODinW/LVIS for detection and grounding, PhraseCut for referring segmentation, VQA on VQA2.0, and COCO captioning.
• Evaluate grounded VQA and grounded image captioning by grounding predicted phrases to image regions during training and inference.

Experimental results

Research questions

• RQ1 Can a single pre-trained model jointly excel at localization and vision-language understanding tasks?
• RQ2 Do additional pre-training losses (inter-image word-region contrastive, MLM) and data scaling improve both localization and VL understanding performance?
• RQ3 How does GLIPv2 perform across diverse datasets and tasks when fine-tuned or evaluated zero-shot?
• RQ4 What is the relative contribution of different pre-training data types (detection, grounding, caption) to downstream VL tasks?

Key findings

• GLIPv2 variants improve zero-shot and fine-tuned performance across COCO, ODinW, and LVIS for detection and grounding.
• Adding inter-image word-region contrastive loss substantially boosts localization and grounding performance.
• MLM loss benefits language understanding tasks such as Flickr30K, VQA, and captioning.
• Scaling up pre-training data with Cap4M and CC/SBU further improves in-domain and domain-gap tasks.
• Grounded image captioning results show competitive caption quality with grounding alignment on Flick30K.
• Inference speed comparisons indicate GLIPv2 variants achieve competitive throughput versus MDETR on COCO and related tasks.