[Paper Review] KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation

Yongfei Liu, Chenfei Wu+4 more|arXiv (Cornell University)|Sep 22, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

KD-VLP proposes an end-to-end vision-and-language pretraining framework that enhances multi-modal representation learning by distilling object-level knowledge from external detectors through two novel pretext tasks: object-guided masked vision modeling and phrase-region alignment. The method achieves state-of-the-art or competitive performance across multiple vision-language benchmarks without requiring object detectors during fine-tuning.

ABSTRACT

Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multi-modal representations from large-scale image-text data and to achieve strong performances on a broad scope of vision-language tasks after finetuning. Previous mainstream VLP approaches typically adopt a two-step strategy relying on external object detectors to encode images in a multi-modal Transformer framework, which suffer from restrictive object concept space, limited image context and inefficient computation. In this paper, we propose an object-aware end-to-end VLP framework, which directly feeds image grid features from CNNs into the Transformer and learns the multi-modal representations jointly. More importantly, we propose to perform object knowledge distillation to facilitate learning cross-modal alignment at different semantic levels. To achieve that, we design two novel pretext tasks by taking object features and their semantic labels from external detectors as supervision: 1.) Object-guided masked vision modeling task focuses on enforcing object-aware representation learning in the multi-modal Transformer; 2.) Phrase-region alignment task aims to improve cross-modal alignment by utilizing the similarities between noun phrases and object labels in the linguistic space. Extensive experiments on a wide range of vision-language tasks demonstrate the efficacy of our proposed framework, and we achieve competitive or superior performances over the existing pretraining strategies.

Motivation & Objective

• To overcome the limitations of two-step VLP methods that rely on external object detectors and suffer from restricted object concept space and inefficient computation.
• To enable end-to-end vision-and-language pretraining that directly uses CNN grid features while still capturing object-level semantics.
• To improve cross-modal alignment by distilling semantic and feature-level knowledge from object detectors into the multi-modal Transformer.
• To design pretext tasks that facilitate learning of object-aware representations and phrase-region correspondence in an end-to-end manner.

Proposed method

• Introduces an end-to-end VLP framework that takes image grid features from a CNN backbone and processes them through a multi-modal Transformer, eliminating dependency on external detectors during inference.
• Proposes object-guided masked vision modeling (OMVM), which reconstructs object region features and semantic labels using surrounding visual and textual context.
• Designs a knowledge-guided masking strategy that selects object candidates for reconstruction based on similarity between noun phrases and object labels.
• Introduces phrase-region alignment (PRA), which aligns linguistic phrase embeddings with object region embeddings using similarity scores in the linguistic space.
• Employs a multi-task learning objective combining standard ITM and MLM with OMVM and PRA to jointly optimize multi-modal representation learning.
• Uses off-the-shelf object detectors (e.g., BUTD, Faster R-CNN) to extract object features and labels during pretraining, which are then distilled into the end-to-end model.

Experimental results

Research questions

• RQ1 Can end-to-end VLP models achieve better performance by incorporating object-level knowledge from external detectors without relying on them during inference?
• RQ2 How does object-guided masked vision modeling improve representation learning compared to standard masked feature reconstruction?
• RQ3 To what extent does phrase-region alignment enhance cross-modal alignment between linguistic entities and visual objects?
• RQ4 Does knowledge distillation from richer object concept spaces (e.g., 1600 categories) lead to better downstream performance than smaller ones?

Key findings

• KD-VLP achieves 66.71% accuracy on NLVR2 and 68.19% on VQA-test dev, outperforming SOHO by 1.87% and 1.50% respectively.
• The phrase-region alignment task improves R@1 by 0.78% on MSCOCO-TR and 1.87% on MSCOCO-IR, demonstrating its effectiveness in cross-modal alignment.
• Using BUTD (1600 categories) as the object detector yields better performance than Faster R-CNN on COCO (80 categories), indicating that a larger object concept space enhances pretraining.
• The ablation study confirms that all proposed pretext tasks—MRC, MRFR, and PRA—contribute positively to performance, with the full model outperforming the baseline by 4.58% on NLVR2.
• Visualization shows that the knowledge-guided masking strategy focuses on phrase-relevant image regions, while attention maps more accurately localize words like 'smiling' to corresponding facial regions.
• Performance gains from object knowledge distillation are consistent across model sizes, with larger models showing greater capacity to leverage external object semantics.