[Paper Review] VisualBERT: A Simple and Performant Baseline for Vision and Language

Liunian Harold Li, Mark Yatskar+3 more|arXiv (Cornell University)|Aug 9, 2019

Multimodal Machine Learning ApplicationsComputer Science37 references1,231 citations

TL;DR

VisualBERT is a simple Transformer-based model that jointly encodes text and image regions for multiple vision-and-language tasks, using pre-training on COCO captions and two visually-grounded language objectives to achieve competitive results.

ABSTRACT

We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks. VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention. We further propose two visually-grounded language model objectives for pre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2, and Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly simpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.

Motivation & Objective

• Motivate a simple, flexible model that jointly reasons about vision and language across diverse tasks (VQA, VCR, NLVR 2, Flickr30K).
• Leverage Transformer self-attention to implicitly align language tokens with image region proposals without heavy task-specific architectures.
• Demonstrate that task-agnostic pre-training on image-caption data improves transfer to downstream V+L tasks.

Proposed method

• Extend BERT with visual embeddings for image regions derived from object detectors.
• Concatenate text embeddings with visual embeddings and feed them into a shared Transformer stack for joint processing.
• Pre-train VisualBERT on COCO captions using two objectives: masked language modeling with image context and sentence-image matching.
• Fine-tune end-to-end on downstream vision-and-language tasks, with optional task-specific pre-training on task data.
• Compare variants including VisualBERT without early fusion and without COCO pre-training to assess component contributions.

Experimental results

Research questions

• RQ1 Can a single, Transformer-based architecture jointly model vision and language across multiple V+L tasks?
• RQ2 Does early fusion of visual and textual features across all Transformer layers improve performance?
• RQ3 How important is task-agnostic pre-training on image-caption data for transferring to downstream tasks?
• RQ4 To what extent do attention heads ground language to image regions and reflect syntactic relations?

Key findings

• VisualBERT achieves competitive or superior performance on VQA 2.0, VCR, NLVR 2, and Flickr30K grounding, often surpassing more complex baselines.
• Early fusion of vision and language (interaction across Transformer layers) is crucial for strong performance.
• Task-agnostic pre-training on COCO captions significantly boosts results; skipping COCO pre-training degrades performance.
• VisualBERT can ground language to image regions without explicit supervision, and certain attention heads track syntactic dependencies (e.g., verbs with their arguments).
• Qualitative analyses show that attention patterns refine alignments across layers and can resolve ambiguous groundings over time.