[Paper Review] ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Jiasen Lu, Dhruv Batra+2 more|arXiv (Cornell University)|Aug 6, 2019

Multimodal Machine Learning Applications30 references1,673 citations

TL;DR

ViLBERT introduces a co-attentional two-stream model that pretrains visio-linguistic representations on large caption-based data and transfers to diverse vision-and-language tasks, achieving state-of-the-art results.

ABSTRACT

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.

Motivation & Objective

• Motivate a single, task-agnostic pretraining approach for vision-and-language (V+L) tasks that can transfer to diverse downstream tasks.
• Develop a co-attentional two-stream architecture that effectively fuses visual and textual information.
• Demonstrate that pretraining on large caption-like data benefits VQA, VCR, and referring expression tasks beyond captioning alone.

Proposed method

• Introduce a co-attentional two-stream model that processes visual and textual inputs with cross-modal attention.
• Pretrain the model on large-scale visio-linguistic data to learn grounded representations that generalize across V+L tasks.
• Ablate pretraining components (e.g., masking loss, alignment loss, co-attention) to assess their impact on downstream tasks.
• Compare against baselines and discuss transfer from caption-style data to non-caption-style tasks (VQA, VCR, RefCOCO+).
• Visualize attention patterns to analyze grounding behavior across layers and attention directions.

Experimental results

Research questions

• RQ1 Can a single visio-linguistic pretraining objective yield representations that transfer effectively to multiple V+L tasks without task-specific heads?
• RQ2 Does a co-attentional two-stream architecture outperform extensions of single-modal models (e.g., BERT) for vision-and-language tasks?
• RQ3 What is the impact of different pretraining components (masking, alignment, co-attention) on downstream V+L performance?
• RQ4 How does pretraining on large caption-like data (Conceptual Captions) influence performance on VQA, RefCOCO+, and VCR compared to non-pretraining or w/o grounding pretraining?
• RQ5 What are the grounding and attention grounding properties of the model across layers and modalities?]
• RQ6 key_findings':['Achieved improved performance on vision-and-language tasks and reportedly surpassed a recent VQA challenge winner, indicating strong state-of-the-art potential.
• RQ7 Ablations show that removing masking, alignment, or co-attention degrades downstream task performance, with masking loss being particularly critical.
• RQ8 Pretraining on Conceptual Captions enables transfer to V+L tasks beyond caption-based retrieval, despite domain differences between CC and downstream tasks.
• RQ9 Visualization indicates that image-to-text co-attention tends to ground early in layers while text-to-image co-attention grounds more with early layers and broader later layers.
• RQ10 The model demonstrates notable gains on VQA and RefCOCO+ in full pretraining versus w/o pretraining configurations, supporting the effectiveness of visio-linguistic pretraining.

Key findings

• Improved results with refined learning rate schedules leading to strong VQA performance.
• Ablations show significant performance drops when removing core pretraining components (masking, align, co-attn).
• Conceptual Captions-based pretraining can transfer to VQA, RefCOCO+, and VCR tasks, with grounding-focused benefits.
• Attention visualizations reveal grounding behavior consistent with cross-modal processing across layers.