[Paper Review] LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Hao Tan, Mohit Bansal|arXiv (Cornell University)|Aug 20, 2019

Multimodal Machine Learning Applications44 references223 citations

TL;DR

LXMERT introduces a three-encoder transformer model to learn vision-language representations, pre-trained with five multimodal tasks, achieving state-of-the-art on VQA and GQA and notable gains on NLVR2.

ABSTRACT

Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results; and also present several attention visualizations for the different encoders. Code and pre-trained models publicly available at: https://github.com/airsplay/lxmert

Motivation & Objective

• Motivate vision-and-language reasoning by modeling visual concepts, language semantics, and their cross-modal alignments.
• Propose a cross-modality Transformer architecture with dedicated encoders for language, object relationships, and joint reasoning.
• Pre-train on large image-sentence corpora using diverse multimodal tasks to capture intra- and cross-modality dependencies.
• Demonstrate state-of-the-art performance on VQA and GQA, and show generalization to NLVR2 through fine-tuning and ablations.

Proposed method

• Three encoders: a language encoder, an object-relationship encoder, and a cross-modality encoder built with self-attention and cross-attention layers.
• Input embeddings that combine word-level sentence representations with position-aware object RoI embeddings from a detected-object detector.
• Five pre-training tasks: (i) masked cross-modality language modeling, (ii) RoI-feature regression for masked objects, (iii) detected-label classification for masked objects, (iv) cross-modality matching, (v) image question answering (QA).
• Cross-modality attentions enable bidirectional information exchange between language and vision, with a dedicated cross-modality encoder stacking multiple layers.
• Training on a large mixture of image-sentence data (9.18M pairs, 100M words, 6.5M objects) and maintaining a fixed object count (36 per image) for efficiency.
• Fine-tuning from pre-trained weights on VQA, GQA, and NLVR2 datasets to evaluate generalization and task adaptation.

Experimental results

Research questions

• RQ1 How can a Transformer-based architecture be designed to jointly model vision and language with explicit cross-modal interactions?
• RQ2 What pre-training objectives best capture intra-modality and cross-modality relationships for vision-language tasks?
• RQ3 To what extent can a cross-modality pre-trained model improve VQA, GQA, and NLVR2 performance compared to single-modality or language-focused pre-training?
• RQ4 How do ablations of model components and pre-training tasks affect downstream vision-language reasoning performance?

Key findings

• LXMERT achieves state-of-the-art results on VQA and GQA across standard metrics.
• On NLVR2, fine-tuning yields a large absolute improvement of 22 percentage points (54% to 76% accuracy).
• Ablation studies show both the novel model components (object-relationship and cross-modality encoders) and the diverse pre-training tasks significantly contribute to gains.
• Cross-modality pre-training without image QA tasks underperforms, highlighting the benefit of image-question answering data for vision-language representations.
• Visualizations of attention in language, object-relationship, and cross-modality encoders illustrate how the model connects text and visual elements.