[Paper Review] Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization

Yang Jin, Kun Xu+14 more|arXiv (Cornell University)|Sep 9, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper proposes LaVIT, a unified language-vision foundation model that uses a dynamic discrete visual tokenizer to convert images into sequence of discrete tokens, enabling end-to-end autoregressive pretraining alongside text under a single generative objective. The method achieves state-of-the-art performance across vision-language benchmarks by treating images as a 'foreign language' readable by LLMs, with dynamic tokenization reducing computation by 36% and improving accuracy over fixed-length baselines.

ABSTRACT

Recently, the remarkable advance of the Large Language Model (LLM) has inspired researchers to transfer its extraordinary reasoning capability to both vision and language data. However, the prevailing approaches primarily regard the visual input as a prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by a frozen LLM. Such an inequitable treatment of vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in a unified form. Specifically, we introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. Coped with this tokenizer, the presented foundation model called LaVIT can handle both image and text indiscriminately under the same generative learning paradigm. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously. Extensive experiments further showcase that it outperforms the existing models by a large margin on massive vision-language tasks. Our code and models are available at https://github.com/jy0205/LaVIT.

Motivation & Objective

• To overcome the inequitable treatment of vision and language in existing MLLMs, where vision is treated merely as a prompt for text generation.
• To enable joint multimodal pretraining by representing visual inputs in a discrete, tokenized format compatible with LLMs’ autoregressive next-token prediction.
• To reduce computational cost and redundancy by dynamically allocating visual tokens based on image complexity and semantic informativeness.
• To demonstrate that unified generative pretraining on discrete visual tokens outperforms regression-based or adapter-based approaches in multimodal understanding and generation.

Proposed method

• Introduces a dynamic visual tokenizer that converts image patches into discrete visual tokens using a learnable codebook and a patch selector.
• Employs a two-stage mechanism: first, a selector identifies the most semantically informative patches from the image; second, a merger compresses them into a variable-length sequence of discrete tokens.
• Uses cross-entropy loss to predict the next discrete visual token in an autoregressive manner, aligning the optimization objective with that of text tokens.
• Leverages a pre-trained vision encoder (e.g., CLIP ViT-L/14) to extract patch features before tokenization, ensuring high-level semantic representation.
• Trains the entire model end-to-end using a unified next-token prediction objective over both text and visual tokens.
• The learned codebook produces semantically meaningful visual tokens, such as representing a skateboard part or giraffe texture, enhancing interpretability.

Experimental results

Research questions

• RQ1 Can visual inputs be effectively represented as discrete tokens compatible with LLMs’ autoregressive training paradigm?
• RQ2 Does dynamic token allocation improve efficiency and performance compared to fixed-length visual tokenization?
• RQ3 Can a unified next-token prediction objective over both vision and language outperform adapter-based or regression-based MLLM approaches?
• RQ4 To what extent does discrete visual tokenization enhance multimodal understanding and generation compared to continuous feature regression?
• RQ5 How does the interpretability and semantic coherence of learned visual codes compare to standard ViT-style patch representations?

Key findings

• LaVIT achieves 74.0 FID on Flickr, 57.7 on VQAv2, and 47.6 on OKVQA, outperforming fixed-length baselines by 2.9, 1.2, and 1.2 points respectively.
• The dynamic tokenization strategy reduces average visual token count from 256 to 94, cutting training time by 40% and inference cost significantly.
• Replacing discrete token prediction with visual feature regression degrades performance, proving the superiority of discrete tokenization for unified learning.
• The visual tokenizer dynamically selects informative patches, filtering out redundant or background content, improving efficiency and representation quality.
• The learned codebook produces semantically meaningful visual tokens—e.g., code 4107 represents a skateboard part and 9146 a giraffe texture—demonstrating interpretability.
• LaVIT generates complex image edits (e.g., dog portrait with cat whiskers) without fine-tuning, showcasing strong zero-shot multimodal generation capability.