[Paper Review] PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

Yuan Yao, Qianyu Chen+5 more|arXiv (Cornell University)|May 23, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

PEVL proposes a unified language modeling framework that integrates discrete object positions with text tokens for detector-free vision-language pre-training, enabling explicit object position modeling through a generalized masked language modeling (GMLM) objective and flexible prompt tuning. It achieves state-of-the-art performance on position-sensitive tasks like referring expression comprehension and phrase grounding, while also improving performance on position-insensitive tasks with grounded inputs.

ABSTRACT

Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mainstream due to their superior computation efficiency and competitive performance. However, the removal of object detectors also deprives the capability of VLP models in explicit object modeling, which is essential to various position-sensitive vision-language (VL) tasks, such as referring expression comprehension and visual commonsense reasoning. To address the challenge, we introduce PEVL that enhances the pre-training and prompt tuning of VLP models with explicit object position modeling. Specifically, PEVL reformulates discretized object positions and language in a unified language modeling framework, which facilitates explicit VL alignment during pre-training, and also enables flexible prompt tuning for various downstream tasks. We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs. We make the data and code for this paper publicly available at https://github.com/thunlp/PEVL.

Motivation & Objective

• To address the lack of explicit object position modeling in detector-free vision-language pre-training (VLP) models, which hinders performance on position-sensitive tasks.
• To unify visual object positions and language tokens within a single language modeling framework to enable joint pre-training and flexible prompt tuning.
• To mitigate the pre-training and fine-tuning gap by reusing the same GMLM head for prompt tuning across diverse downstream tasks.
• To improve position learning robustness by introducing an ordering-aware objective for masked position reconstruction, reducing sensitivity to bounding box annotation noise.

Proposed method

• Discretize object bounding box coordinates into discrete position tokens and concatenate them after corresponding object description tokens in the input sequence.
• Formulate pre-training as a generalized masked language modeling (GMLM) task, where the model predicts both masked text and position tokens from cross-modal context.
• Introduce an ordering-aware objective for position token reconstruction that assigns higher soft labels to nearby positions, preserving spatial ordering despite discretization.
• Enable prompt tuning by reformulating downstream VL tasks as fill-in-the-blank problems using the same GMLM head, minimizing the pre-training-fine-tuning distribution shift.
• Use a shared Transformer encoder-decoder architecture to maintain consistency between pre-training and prompt tuning, supporting both position-input and position-output tasks.
• Maintain identical masking ratios for text and position tokens (as in BERT) to ensure balanced learning during pre-training.

Experimental results

Research questions

• RQ1 Can explicit object position modeling be effectively integrated into detector-free vision-language pre-training without relying on object detectors?
• RQ2 How can discrete position tokens be modeled in a way that preserves spatial ordering and is robust to annotation noise?
• RQ3 Can a unified GMLM framework jointly pre-train language and position representations while enabling flexible prompt tuning across diverse vision-language tasks?
• RQ4 Does the proposed ordering-aware objective for position reconstruction lead to improved localization and reasoning performance compared to standard masking?
• RQ5 Can the same GMLM head be effectively reused for prompt tuning across both position-sensitive and position-insensitive tasks, reducing the pre-training-fine-tuning gap?

Key findings

• PEVL achieves state-of-the-art performance on referring expression comprehension and phrase grounding tasks among detector-free VLP models, outperforming prior methods that lack explicit position modeling.
• The proposed ordering-aware objective for masked position reconstruction leads to faster convergence and better performance on RefCOCO during intermediate pre-training, as shown in Table 7.
• PEVL improves performance on position-insensitive tasks such as visual question answering when grounded inputs are used, demonstrating enhanced robustness and generalization.
• The model achieves strong localization results on small objects, comparable to regression-based models, as visualized in Figure 2.
• Ablation studies confirm that high masking ratios and comprehensive coverage of masked positions are critical for effective position learning.
• The unified prompt tuning framework with a shared GMLM head enables consistent and effective adaptation across diverse downstream tasks, minimizing distribution shift between pre-training and fine-tuning.