[Paper Review] VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Weijie Su, Xizhou Zhu+5 more|arXiv (Cornell University)|Aug 22, 2019

Multimodal Machine Learning Applications45 references782 citations

TL;DR

VL-BERT introduces a unified visual-linguistic Transformer pre-trained on image-caption data and text corpora, achieving state-of-the-art results on VCR, VQA, and referring expression tasks with a single-model, end-to-end approach.

ABSTRACT

We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{https://github.com/jackroos/VL-BERT}.

Motivation & Objective

• Develop a generic, pre-trainable visual-linguistic representation that can be fine-tuned for multiple downstream tasks.
• Integrate visual RoI features and linguistic inputs into a single Transformer backbone with flexible cross-modal attention.
• Pre-train on large-scale visual-linguistic and text-only corpora to align visual and linguistic cues and improve generalization.
• Demonstrate state-of-the-art performance on VCR, VQA, and referring expression comprehension with a single model.

Proposed method

• Extend Transformer architecture to process both word and RoI inputs as a unified sequence.
• Represent inputs with token, visual feature, segment, and position embeddings; include a new visual feature embedding for RoIs.
• Pre-train with two tasks on visual-linguistic data: Masked Language Modeling with Visual Clues and Masked RoI Classification with Linguistic Clues.
• Pre-train on Conceptual Captions (visual-linguistic) and BooksCorpus/Wikipedia (text corpora) with a 1:1 sampling mix.
• Fine-tune end-to-end for downstream tasks using task-specific input/output formats (e.g., <Question, Answer, Image>, <Caption, Image>).

Experimental results

Research questions

• RQ1 Can a single, unified Transformer-based model effectively learn and align visual and linguistic representations across multiple tasks?
• RQ2 Does joint pre-training on visual-linguistic and text-only data improve performance on downstream visual-linguistic tasks compared to single-domain pre-training?
• RQ3 What is the impact of incorporating visual clues into MLM and RoI classification on downstream tasks like VCR, VQA, and RefCOCO+?
• RQ4 Can a pre-trained VL-BERT model achieve state-of-the-art results with a single model architecture on diverse benchmarks?

Key findings

• VL-BERT achieves strong performance across multiple visual-linguistic tasks with a single unified model.
• Pre-training on visual-linguistic data provides about 1.0 percentage point improvement on the final VCR task (Q→AR) compared to non-pre-trained baselines.
• VL-BERT LARGE attains competitive results: VCR val Q→A 75.5, QA→R 75.8; test Q→A 77.9, test QA→R 78.4; RefCOCO+ val 80.31, testA 83.62, testB 75.45; VQA test-dev 71.79, test-std 72.22.
• On VQA, VL-BERT BASE/LARGE outperform non-pre-trained baselines and surpass some concurrent methods in single-model settings (e.g., Large achieves 71.79 on test-dev and 72.22 on test-std).
• On RefCOCO+, VL-BERT LARGE yields strong results (testA 83.62, testB 62.30 with detected regions).
• VL-BERT demonstrates state-of-the-art performance for visual commonsense reasoning (VCR) among single-model approaches at the time of publication.