[Paper Review] VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation

Jinguo Zhu, Xiaohan Ding+6 more|arXiv (Cornell University)|Dec 14, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

VL-GPT is a unified generative pre-trained transformer that enables end-to-end multimodal understanding and generation using a single auto-regressive objective. By introducing a learnable image tokenizer-detokenizer framework, it maps raw images to continuous visual embeddings and back, allowing interleaved image-text sequences to be processed and generated uniformly, achieving strong zero-shot and few-shot performance across vision-language tasks.

ABSTRACT

In this work, we introduce Vision-Language Generative Pre-trained Transformer (VL-GPT), a transformer model proficient at concurrently perceiving and generating visual and linguistic data. VL-GPT achieves a unified pre-training approach for both image and text modalities by employing a straightforward auto-regressive objective, thereby enabling the model to process image and text as seamlessly as a language model processes text. To accomplish this, we initially propose a novel image tokenizer-detokenizer framework for visual data, specifically designed to transform raw images into a sequence of continuous embeddings and reconstruct them accordingly. In combination with the existing text tokenizer and detokenizer, this framework allows for the encoding of interleaved image-text data into a multimodal sequence, which can subsequently be fed into the transformer model. Consequently, VL-GPT can perform large-scale pre-training on multimodal corpora utilizing a unified auto-regressive objective (i.e., next-token prediction). Upon completion of pre-training, VL-GPT exhibits remarkable zero-shot and few-shot performance across a diverse range of vision and language understanding and generation tasks, including image captioning, visual question answering, text-to-image generation, and more. Additionally, the pre-trained model retrains in-context learning capabilities when provided with multimodal prompts. We further conduct instruction tuning on our VL-GPT, highlighting its exceptional potential for multimodal assistance. The source code and model weights shall be released.

Motivation & Objective

• To unify vision and language pre-training under a single auto-regressive objective, eliminating the need for separate modeling of image input and generation.
• To address the inconsistency in image representation between input and output in existing multimodal models by enabling consistent visual embedding modeling.
• To enable end-to-end multimodal generation, including text, images, and interleaved outputs, through a unified sequence modeling framework.
• To explore the potential of in-context learning and instruction tuning in multimodal generative models for general-purpose assistance.
• To develop a tokenizer-detokenizer framework that preserves both image details and semantic content using dual supervision from CLIP and diffusion models.

Proposed method

• Proposes a novel image tokenizer-detokenizer framework that converts raw images into continuous visual embeddings and reconstructs them in pixel space, analogous to text tokenization in LLMs.
• Uses pre-trained CLIP image and caption embeddings as supervision to train the tokenizer and detokenizer, ensuring rich semantic and perceptual fidelity.
• Employs joint training with weight initialization from pre-trained image encoders and high-quality diffusion models to improve efficiency and performance.
• Encodes interleaved image-text data into a unified multimodal sequence using text BPE tokenization and visual embedding sequences, enabling auto-regressive modeling.
• Applies mixed loss: mean squared error (MSE) for visual embeddings and cross-entropy for text tokens, enabling joint next-token prediction across modalities.
• Utilizes both text and image condition embeddings during detokenization to enhance reconstruction quality and semantic consistency.

Experimental results

Research questions

• RQ1 Can a unified auto-regressive objective be applied to both vision and language modalities using a shared transformer architecture?
• RQ2 How effective is a learnable image tokenizer-detokenizer framework in preserving image details and semantics for end-to-end multimodal generation?
• RQ3 Does joint estimation of text and image condition embeddings improve image reconstruction and downstream multimodal task performance?
• RQ4 Can VL-GPT generalize to zero-shot and few-shot vision-language tasks using in-context learning with multimodal demonstrations?
• RQ5 To what extent does instruction tuning enhance VL-GPT’s capability as a multimodal assistant for diverse generation and understanding tasks?

Key findings

• VL-GPT achieves state-of-the-art zero-shot performance on image captioning, visual question answering, and text-to-image generation tasks.
• On the VQAv2 benchmark, VL-GPT achieves 57.2% accuracy in few-shot setting (k=2), outperforming prior methods under the same few-shot conditions.
• The model demonstrates strong in-context learning: performance on VQA increases with the number of few-shot examples in the prompt, showing positive correlation.
• Ablation studies confirm that using both text and image condition embeddings (eₜ + eᵥ) yields the best image reconstruction quality and downstream performance.
• Qualitative results show VL-GPT generates highly realistic images from long, complex text prompts and maintains multimodal coherence in multi-turn dialogues.
• Instruction tuning enables VL-GPT to function as a versatile multimodal assistant capable of flexible, context-aware generation of text and images.