[Paper Review] UNITER: UNiversal Image-TExt Representation Learning

Yen-Chun Chen, Linjie Li+6 more|arXiv (Cornell University)|Sep 25, 2019

Multimodal Machine Learning ApplicationsComputer Science62 references184 citations

TL;DR

UNITER pre-trains a unified image-text Transformer with conditional masking and Optimal Transport-based Word-Region Alignment, achieving state-of-the-art across six Vision-and-Language tasks.

ABSTRACT

Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are simultaneously processed for joint visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design four pre-training tasks: Masked Language Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text Matching (ITM), and Word-Region Alignment (WRA). Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). In addition to ITM for global image-text alignment, we also propose WRA via the use of Optimal Transport (OT) to explicitly encourage fine-grained alignment between words and image regions during pre-training. Comprehensive analysis shows that both conditional masking and OT-based WRA contribute to better pre-training. We also conduct a thorough ablation study to find an optimal combination of pre-training tasks. Extensive experiments show that UNITER achieves new state of the art across six V+L tasks (over nine datasets), including Visual Question Answering, Image-Text Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR$^2$. Code is available at https://github.com/ChenRocks/UNITER.

Motivation & Objective

• Motivate the need for a universal image-text representation that generalizes across diverse Vision-and-Language (V+L) tasks.
• Develop a large-scale pre-training framework using four image-text datasets to learn joint multimodal embeddings.
• Introduce conditional masking for MLM/MRM and a novel Word-Region Alignment task based on Optimal Transport to improve cross-modal alignment.
• Show that the proposed pre-training yields improvements across multiple downstream V+L benchmarks and analyze task/dataset contributions.

Proposed method

• Encode image regions via Faster R-CNN features and positions; encode text tokens with WordPieces and positional embeddings; fuse with a multi-layer Transformer.
• Pre-train with four tasks: Masked Language Modeling conditioned on image regions (MLM), Masked Region Modeling conditioned on text (MRM) with three variants, Image-Text Matching (ITM), and Word-Region Alignment (WRA).
• Use conditional masking so only one modality is masked at a time during pre-training to prevent misalignment.
• Apply Optimal Transport to learn a transport plan between word embeddings and image region embeddings, minimizing a cost-based OT distance as the WRA loss.
• Explore three MRM variants: Masked Region Feature Regression (MRFR), Masked Region Classification (MRC), and MRC with KL-divergence (MRC-kl); and evaluate combinations of pre-training tasks and data sources.

Experimental results

Research questions

• RQ1 Can a single, large-scale pre-trained model yield strong performance across a range of V+L tasks with joint multimodal embeddings?
• RQ2 Do conditional masking and OT-based Word-Region Alignment improve cross-modal alignment and downstream task performance compared to prior methods?
• RQ3 What is the optimal combination of pre-training tasks and datasets to maximize transfer to diverse V+L benchmarks?
• RQ4 How does increasing pre-training data with in-domain and out-of-domain sources affect downstream results?

Key findings

• UNITER achieves new state-of-the-art results across six V+L tasks on nine datasets.
• Conditional masking and OT-based WRA both contribute to better cross-modal alignment and performance.
• An optimal combination of MLM, ITM, MRM variants (especially MRC-kl and MRFR), and WRA yields the best Meta-Sum improvements.
• Pre-training on larger, more diverse data (in-domain + out-of-domain) further boosts performance, with diminishing gains when data diverges from downstream task distributions.