[Paper Review] MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning

Zejun Li, Zhihao Fan+4 more|arXiv (Cornell University)|Jan 29, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

MVPTR proposes a two-stage vision-language pre-training framework that enhances multi-level semantic alignment by explicitly modeling object-level and phrase-level semantics in vision and language modalities. By introducing masked concept recovering and structured phrase grounding tasks, MVPTR achieves state-of-the-art performance on image-text retrieval, visual grounding, and VQA benchmarks, demonstrating the effectiveness of hierarchical alignment in multi-modal representation learning.

ABSTRACT

Previous vision-language pre-training models mainly construct multi-modal inputs with tokens and objects (pixels) followed by performing cross-modality interaction between them. We argue that the input of only tokens and object features limits high-level semantic alignment like phrase-to-region grounding. Meanwhile, multi-level alignments are inherently consistent and able to facilitate the representation learning synergistically. Therefore, in this paper, we propose to learn Multi-level semantic alignment for Vision-language Pre-TRaining (MVPTR). In MVPTR, we follow the nested structure of both modalities to introduce concepts as high-level semantics. To ease the learning from multi-modal multi-level inputs, our framework is split into two stages, the first stage focuses on intra-modality multi-level representation learning, the second enforces interactions across modalities via both coarse-grained and fine-grained semantic alignment tasks. In addition to the commonly used image-text matching and masked language model tasks, we introduce a masked concept recovering task in the first stage to enhance the concept representation learning, and two more tasks in the second stage to explicitly encourage multi-level alignments across modalities. Our code is available at https://github.com/Junction4Nako/mvp_pytorch.

Motivation & Objective

• To address the limitation of existing vision-language pre-training models that focus only on token-to-region alignment, missing higher-level semantic connections such as phrase-to-region grounding.
• To enable synergistic learning across multiple semantic levels—word, phrase, and object—by structuring inputs with nested semantic concepts in both modalities.
• To improve cross-modal alignment by decoupling intra-modal representation learning from cross-modal interaction through a two-stage training paradigm.
• To explicitly model phrase-level semantics using scene graph-derived concepts and object tags as anchors, enhancing representation quality for downstream tasks.

Proposed method

• Introduces a two-stage training framework: first, uni-modal learning to build multi-level representations using masked concept recovering (MCR) for vision and phrase-aware token generation for language.
• Incorporates object tags and phrase-level concepts (from scene graphs) as high-level semantic units to bridge tokens and visual regions.
• Employs masked concept recovering (MCR) on both vision and language sides to align object tags with visual regions and phrases with related tokens.
• Uses coarse-grained and fine-grained alignment tasks in the second stage to enforce cross-modal interaction at multiple semantic levels.
• Leverages a disentangled attention mechanism to separate intra-modal and cross-modal attention, preventing interference between uni-modal and cross-modal learning.
• Introduces a phrase-level grounding task using weighted phrase grounding (WPG) to explicitly model relationships between phrases and visual regions.

Experimental results

Research questions

• RQ1 Can explicit modeling of multi-level semantics (word, phrase, object) improve vision-language pre-training beyond token-to-region alignment?
• RQ2 How does separating uni-modal and cross-modal learning stages affect the quality of multi-level representation learning?
• RQ3 What is the impact of introducing masked concept recovering and phrase grounding tasks on downstream vision-language benchmark performance?
• RQ4 To what extent do object-level and phrase-level semantics serve as foundational components for higher-level alignment in vision-language models?

Key findings

• The ablation study shows that removing object-level concepts significantly degrades performance on image-text matching and phrase grounding, confirming their foundational role in multi-level alignment.
• Explicit modeling of phrase-level semantics via WPG and MCR improves visual grounding and fine-grained matching by 2.1% and 1.8% respectively, demonstrating the benefit of structured phrase representation.
• The two-stage framework outperforms end-to-end models by reducing attention interference, achieving SOTA results on MSCOCO, Flickr30k, VQA, and SNLI-VE benchmarks.
• Visualization confirms that the model learns to align phrase concepts (e.g., 'group-of-zebra') with corresponding visual regions, validating the effectiveness of phrase-level attention mechanisms.
• The model generalizes well across datasets, with phrases like 'two-man' and 'young-boy' appearing frequently and consistently across MSCOCO, Flickr30k, and VQA.
• The use of scene graph-derived concepts leads to more robust phrase representations, as shown in qualitative attention maps where phrase nodes correctly link to relevant image regions.