[Paper Review] VinVL: Revisiting Visual Representations in Vision-Language Models

Pengchuan Zhang, Xiujun Li+6 more|arXiv (Cornell University)|Jan 2, 2021

Multimodal Machine Learning ApplicationsComputer Science42 references60 citations

TL;DR

The authors develop a large, object-centric visual detector trained on multiple datasets to produce richer visual features, and integrate it with an enhanced Oscar+ VL pre-training pipeline, achieving new state-of-the-art results across seven vision-language tasks.

ABSTRACT

This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric representations of images. Compared to the most widely used \emph{bottom-up and top-down} model \cite{anderson2018bottom}, the new model is bigger, better-designed for VL tasks, and pre-trained on much larger training corpora that combine multiple public annotated object detection datasets. Therefore, it can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses mainly on improving the vision-language fusion model and leaves the object detection model improvement untouched, we show that visual features matter significantly in VL models. In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model \oscar \cite{li2020oscar}, and utilize an improved approach \short\ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks. Our results show that the new visual features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. We will release the new object detection model to public.

Motivation & Objective

• Demonstrate that richer visual features significantly impact vision-language performance.
• Develop a large-scale object detection model that covers diverse objects and attributes for VL tasks.
• Pre-train and fine-tune a unified vision-language model (Oscar+) using enhanced visual features to improve multiple VL benchmarks.

Proposed method

• Pre-train a large object detector on a unified corpus combining COCO, OpenImages, Objects365, and Visual Genome to produce 1848 object classes including 524 attributes.
• Inject an attribute branch and fine-tune on Visual Genome to enhance object-attribute detection.
• Use an efficient region feature extractor that speeds up feature extraction for VL tasks.
• Pre-train Oscar+ with a 3-way contrastive loss that aligns captions/QA with image tags and regions.
• Fine-tune Oscar+ on seven VL tasks including VQA, GQA, NLVR2, image captioning, NoCaps, image/text retrieval.

Experimental results

Research questions

• RQ1 Does improving the quality and diversity of visual features enhance performance across vision-language tasks?
• RQ2 Can a larger, more diverse object-centric detector improve downstream VL understanding and generation tasks when integrated with a Transformer-based VL fusion model?
• RQ3 What design choices in data, model architecture, and pre-training objectives most contribute to VL gains?
• RQ4 How do the new visual features affect performance on both recognition-style tasks (VQA, GQA) and generation/retrieval tasks (captioning, NoCaps, retrieval, NLVR2)?

Key findings

• Replacing the previous OD features with VinVL’s richer region features yields consistent state-of-the-art gains across seven VL tasks.
• VinVL gains are substantial, with analysis attributing about 95% of the overall improvement to visual feature enhancements.
• The new object detector increases coverage of semantically meaningful regions and enriches object concepts and attributes.
• Oscar+ with VinVL achieves new SOTA on VQA, GQA, NLVR2, NoCaps, and retrieval tasks, and competitive results on image captioning.
• Efficient region feature extraction and inclusion of attributes lead to faster inference without accuracy loss.