[Paper Review] Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Gen Li, Nan Duan+4 more|arXiv (Cornell University)|Aug 16, 2019

Multimodal Machine Learning ApplicationsComputer Science40 references117 citations

TL;DR

Unicoder-VL pre-trains a multi-layer Transformer to learn joint vision-language representations using three cross-modal objectives, enabling strong image-text retrieval and competitive visual commonsense reasoning after fine-tuning.

ABSTRACT

We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM and Unicoder, both visual and linguistic contents are fed into a multi-layer Transformer for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling (MLM), Masked Object Classification (MOC) and Visual-linguistic Matching (VLM). The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly. The last task tries to predict whether an image and a text describe each other. After pretraining on large-scale image-caption pairs, we transfer Unicoder-VL to caption-based image-text retrieval and visual commonsense reasoning, with just one additional output layer. We achieve state-of-the-art or comparable results on both two tasks and show the powerful ability of the cross-modal pre-training.

Motivation & Objective

• Motivate a universal cross-modal encoder that handles long linguistic sequences with visual context.
• Leverage large-scale image-caption data to learn joint representations through cross-modal pre-training.
• Design and evaluate three cross-modal pre-training tasks to align visual and linguistic modalities.

Proposed method

• Use a multi-layer Transformer initialized from BERT to fuse visual region features and linguistic tokens.
• Inject image region embeddings and location features, jointly encoded with text tokens.
• Pre-train with three objectives: Masked Language Modeling (MLM), Masked Object Classification (MOC), and Visual-linguistic Matching (VLM).
• MLM predicts masked words using surrounding text and all image regions.
• MOC predicts object categories for masked visual regions.
• VLM trains a binary predictor to decide if an image-text pair describes each other.

Experimental results

Research questions

• RQ1 Can a single Transformer-based encoder learn robust cross-modal representations from image-caption data?
• RQ2 Do cross-modal pre-training objectives improve downstream image-text retrieval and visual commonsense reasoning?
• RQ3 How does model scale and pre-training data size affect cross-modal transfer performance?

Key findings

• Pre-trained Unicoder-VL achieves state-of-the-art results on image-text retrieval benchmarks after fine-tuning on MSCOCO and Flickr30K.
• Zero-shot retrieval with Unicoder-VL shows general cross-modal grounding without task-specific fine-tuning.
• Unicoder-VL delivers competitive results on Visual Commonsense Reasoning (VCR), indicating benefits of cross-modal pre-training for cognitive tasks.
• Model performance improves with larger Transformer depth and more pre-training data.