[Paper Review] ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Wonjae Kim, Bokyung Son+1 more|arXiv (Cornell University)|Feb 5, 2021

Multimodal Machine Learning Applications62 references538 citations

TL;DR

ViLT presents a minimal vision-and-language pre-training model that uses patch-projected image tokens and a single transformer for multimodal interaction, achieving competitive performance with much faster inference by removing region-based or convolutional visual embedders.

ABSTRACT

Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.

Motivation & Objective

• Motivate reducing reliance on heavy visual embedders and region supervision in Vision-and-Language Pre-training (VLP).
• Propose a minimal, convolution-free ViLT architecture that processes images as patches alongside text in a single transformer.
• Demonstrate efficiency gains (faster runtime, fewer parameters) while maintaining competitive downstream performance.

Proposed method

• Introduce ViLT with patch-projected image embeddings fed directly into a ViT-inspired transformer without convolutional backbones or region features.
• Use two pre-training objectives common in VLP: image-text matching (ITM) and masked language modeling (MLM).
• Employ whole word masking (WWM) for MLM to improve cross-modal learning.
• Incorporate an image augmentation (RandAugment) during fine-tuning to boost generalization.
• Optional: introduce word-patch alignment (WPA) via IPOT-based optimal transport to align textual tokens with visual patches.

Experimental results

Research questions

• RQ1 Can a vision-and-language model operate without convolutional visual components or region-based supervision and still perform well on standard VLP tasks?
• RQ2 How do pre-training strategies like whole word masking and image augmentation affect cross-modal performance in a minimal ViLT setup?
• RQ3 What is the computational efficiency gain of a convolution-free, patch-based visual embedding compared to region-feature or grid-feature VLP models?

Key findings

• ViLT achieves up to tens of times faster inference than region-feature-based models and at least four times faster than grid-feature models while maintaining competitive or better downstream performance.
• ViLT performs competitively on VQA, NLVR2, and retrieval benchmarks despite omitting heavy visual embedders.
• Whole word masking and image augmentations significantly improve downstream performance in ViLT.
• Training longer steps and using RandAugment during fine-tuning generally improve results, while adding an extra MRM-like objective (MPP) showed no benefit in ablations.