[Paper Review] Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

Junnan Li, Ramprasaath R. Selvaraju+4 more|arXiv (Cornell University)|Jul 16, 2021

Multimodal Machine Learning Applications60 references822 citations

TL;DR

ALBEF introduces an image-text contrastive alignment before fusion and momentum distillation to leverage noisy web data, achieving state-of-the-art results on multiple vision-language tasks without bounding boxes. It jointly learns unimodal and multimodal representations and uses a momentum teacher to generate pseudo-targets for self-training.

ABSTRACT

Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. In this paper, we introduce a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. Unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. In order to improve learning from noisy web data, we propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. We provide a theoretical analysis of ALBEF from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. ALBEF achieves state-of-the-art performance on multiple downstream vision-language tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained on orders of magnitude larger datasets. On VQA and NLVR$^2$, ALBEF achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-art, while enjoying faster inference speed. Code and pre-trained models are available at https://github.com/salesforce/ALBEF/.

Motivation & Objective

• Motivate and develop a detector-free vision-language pre-training framework that aligns image and text representations before fusion.
• Propose an intermediate image-text contrastive loss (ITC) to ground unimodal encoders and facilitate cross-modal learning.
• Introduce Momentum Distillation (MoD) to improve learning from noisy web data by using a momentum-averaged teacher to generate pseudo-targets.
• Demonstrate that ALBEF learns robust vision-language representations that perform well on retrieval, VQA, NLVR2, VE, and weakly-supervised grounding.
• Provide theoretical framing via mutual information maximization to justify design choices.

Proposed method

• Encode images with a detector-free ViT-based image encoder and texts with a transformer-based text encoder.
• Fuse image and text through a 6-layer multimodal transformer with cross-modal attention at each layer.
• Apply an image-text contrastive loss (ITC) on unimodal representations to align image and text before fusion.
• Use in-batch hard negative mining to strengthen image-text matching (ITM) on the multimodal encoder.
• Train with MLM and ITM losses to learn multimodal interactions; optimize a joint loss L = L_itc + L_mlm + L_itm.
• Employ Momentum Distillation (MoD): maintain a momentum model to generate pseudo-targets for ITC and MLM losses, and mix with original losses using a weighting α (0.4) to improve learning and downstream performance.

Experimental results

Research questions

• RQ1 Can a detector-free VLP approach with an intermediate ITC loss before fusion improve cross-modal interaction learning?
• RQ2 Does momentum distillation enable effective learning from noisy web-scale vision-language data without explicit bounding boxes?
• RQ3 How do ITC, MLM, ITM, and MoD interact under a mutual-information maximization view to improve vision-language representations?
• RQ4 What performance gains do ALBEF variants achieve on image-text retrieval, VQA, NLVR2, visual entailment, and weakly-supervised grounding compared to state-of-the-art methods?

Key findings

• ALBEF achieves state-of-the-art performance on image-text retrieval, outperforming methods trained on much larger data.
• With detector-free inputs, ALBEF achieves competitive or superior results on VQA, NLVR2, and VE tasks, and offers faster inference than detector-based methods.
• Momentum Distillation (MoD) improves pre-training and downstream tasks, enabling learning from larger, noisier web data.
• ALBEF with ITC, MLM, ITM and MoD shows substantial gains over baselines such as MLM+ITM and hard-negative ITM variants across multiple tasks.
• A mutual-information perspective explains ALBEF components as generating diverse views to maximize MI between image-text representations.