[Paper Review] Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

Yan Zeng, Xinsong Zhang+1 more|arXiv (Cornell University)|Nov 16, 2021

Multimodal Machine Learning ApplicationsComputer Science34 references96 citations

TL;DR

X-VLM presents multi-grained vision-language pre-training by aligning texts with visual concepts (objects, regions, images) without bounding-box input at inference, achieving state-of-the-art on multiple V+L tasks.

ABSTRACT

Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.

Motivation & Objective

• Motivate learning vision-language alignments beyond object-centric or image-wide representations.
• Enable multi-granularity alignments by associating texts with visual concepts at object, region, and image levels.
• Develop a training objective that jointly locates visual concepts and aligns them with textual descriptions.
• Demonstrate effectiveness across downstream V+L tasks without bounding box inputs at inference.

Proposed method

• Represent visual concepts as multi-grained outputs from a vision transformer, including objects, regions, and the full image.
• Formulate pre-training data so each image has multiple bounding boxes each linked to a text description of the associated concept.
• Optimize with bounding box prediction (box regression + IoU) and multi-task losses for alignment: contrastive loss, matching loss, and masked language modeling loss.
• Use cross-modal transformer to fuse vision and language features via cross-attention at each layer.
• Train with 4M and 16M image settings to evaluate data efficiency and scalability, using AdamW optimization and scheduled learning rate warm-up.
• Avoid bounding-box annotations at inference; leverage learned multi-granular alignments for downstream tasks.

Experimental results

Research questions

• RQ1 Can multi-grained visual concepts (objects, regions, and image-level concepts) be aligned with text descriptions during pre-training?
• RQ2 Does jointly locating visual concepts and aligning them to text improve performance on retrieval, reasoning, grounding, and captioning tasks?
• RQ3 How does X-VLM perform compared to object-centric and coarse-grained V+L models across standard benchmarks?
• RQ4 Is it feasible to achieve strong V+L performance with moderate model size and training data?

Key findings

• X-VLM outperforms state-of-the-art methods on image-text retrieval (MSCOCO and Flickr30K) in both 4M and 16M pre-training settings.
• On MSCOCO, X-VLM (4M) achieves 80.4/95.5/98.2 for TR and 63.1/85.7/91.6 for IR, while X-VLM (16M) achieves 81.2/95.6/98.2 for TR and 63.4/85.8/91.5 for IR.
• X-VLM shows improvements in visual reasoning (VQA and NLVR2) and visual grounding (RefCOCO+) over VinVL and other baselines, with particular gains in grounding (4.5% absolute over UNITER on RefCOCO+).
• X-VLM delivers competitive image captioning results comparable to SoTA generative methods, with CIDEr improvements observed after fine-tuning.
• Ablation shows that removing object or region concepts or the bbox loss degrades performance, highlighting the importance of multi-grained learning.