[Paper Review] An Introduction to Vision-Language Modeling

Florian Bordes, Richard Yuanzhe Pang+39 more|arXiv (Cornell University)|May 27, 2024

Geographic Information Systems Studies33 citations

TL;DR

This paper provides an accessible overview of Vision-Language Models (VLMs), detailing training paradigms (contrastive, masking, generative, and pretrained-backbone approaches), data and evaluation practices, and extensions to video.

ABSTRACT

Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.

Motivation & Objective

• Explain what VLMs are and why they are important for bridging vision and language.
• Categorize VLMs into meaningful families based on training objectives and architectures.
• Guide researchers on data, training practices, and evaluation methodologies for VLMs.
• Discuss extensions to video and considerations for responsible evaluation and grounding.

Proposed method

• Classify VLM approaches into four families: contrastive, masking, generative, and pretrained-backbone models.
• Describe key models exemplifying each paradigm (e.g., CLIP for contrastive, FLAVA/MaskVLM for masking, CoCa/CM3leon for generative, Frozen/MiniGPT for backbones).
• Explain the information-theoretic view of VLM objectives and its relation to rate-distortion thinking.
• Outline practical training recipes including data curation, augmentation, and grounding techniques.
• Discuss evaluation strategies, biases, memorization, hallucinations, and red-teaming for responsible VLM assessment.

Experimental results

Research questions

• RQ1 What are the main training paradigms used to build vision-language models and how do they differ?
• RQ2 How can VLMs be trained efficiently with various data and backbone strategies?
• RQ3 What are robust evaluation practices to judge visio-linguistic abilities, biases, and reliability in VLMs?
• RQ4 How can VLMs be extended from images to videos and what challenges arise?

Key findings

• Contrastive, masking, and generative objectives can be combined in VLMs, and their choice depends on task and resources.
• Pretrained backbones (e.g., LLMs) can drastically reduce training costs while enabling strong multimodal capabilities.
• An information-theoretic lens connects masking and contrastive objectives as rate-distortion trade-offs in representation learning.
• Generative models enable discriminative tasks via conditional likelihood and offer robustness and compositional reasoning advantages.
• Evaluation of VLMs must address bias, hallucinations, memorization, and the impact of training data on downstream performance.