[Paper Review] Autoregressive Image Generation without Vector Quantization

Tianhong Li, Yonglong Tian+3 more|arXiv (Cornell University)|Jun 17, 2024

Image Retrieval and Classification Techniques4 citations

TL;DR

This paper proposes a novel Diffusion Loss for autoregressive image generation that eliminates the need for vector quantization by modeling per-token probability distributions through a continuous-valued diffusion process. By training a small denoising network conditioned on autoregressively predicted vectors, the method achieves state-of-the-art FID scores (1.55) with inference speeds under 0.3 seconds per image on ImageNet 256×256, demonstrating that autoregressive models can be effectively applied in continuous spaces without discrete tokenization.

ABSTRACT

Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens. We observe that while a discrete-valued space can facilitate representing a categorical distribution, it is not a necessity for autoregressive modeling. In this work, we propose to model the per-token probability distribution using a diffusion procedure, which allows us to apply autoregressive models in a continuous-valued space. Rather than using categorical cross-entropy loss, we define a Diffusion Loss function to model the per-token probability. This approach eliminates the need for discrete-valued tokenizers. We evaluate its effectiveness across a wide range of cases, including standard autoregressive models and generalized masked autoregressive (MAR) variants. By removing vector quantization, our image generator achieves strong results while enjoying the speed advantage of sequence modeling. We hope this work will motivate the use of autoregressive generation in other continuous-valued domains and applications. Code is available at: https://github.com/LTH14/mar.

Motivation & Objective

• To challenge the conventional belief that autoregressive models require discrete, vector-quantized representations for image generation.
• To develop a continuous-valued alternative to vector quantization that maintains the benefits of autoregressive modeling.
• To unify standard autoregressive and masked autoregressive models under a generalized framework compatible with continuous-valued tokens.
• To improve image generation quality and speed by replacing categorical cross-entropy with a diffusion-based probability modeling approach.
• To demonstrate the effectiveness of the proposed method across diverse architectures and tokenization strategies.

Proposed method

• The method replaces traditional categorical cross-entropy loss with a Diffusion Loss that models the per-token probability distribution using a denoising diffusion process.
• For each autoregressively predicted token vector z, a small denoising network (e.g., an MLP) is trained to model the distribution p(x|z) of the actual image token x.
• The denoising network is trained jointly with the autoregressive model via backpropagation, enabling end-to-end optimization of the continuous-valued token prediction.
• The approach is compatible with both standard autoregressive (AR) and masked autoregressive (MAR) models, allowing flexible generation order and multi-token prediction per step.
• The method supports both pixel-based and continuous-valued tokenization, and can be applied with or without classifier-free guidance (CFG).
• Inference involves sampling from the reverse diffusion process conditioned on the predicted z vector, enabling high-fidelity image generation.

Experimental results

Research questions

• RQ1 Can autoregressive image generation be effectively performed without vector quantization by modeling per-token distributions through diffusion?
• RQ2 Does replacing categorical cross-entropy with a diffusion-based loss improve image generation quality and speed in continuous-valued spaces?
• RQ3 Can the proposed Diffusion Loss be generalized across different autoregressive architectures, including masked autoregressive variants?
• RQ4 How does the performance of the proposed method compare to state-of-the-art models in terms of FID, IS, and inference speed?
• RQ5 What is the impact of architectural choices such as random-order masking and bidirectional attention on the quality and efficiency of continuous-valued autoregressive generation?

Key findings

• The proposed Diffusion Loss achieves a Fréchet Inception Distance (FID) of 1.55 on ImageNet 256×256, outperforming previous token-based methods and approaching the performance of leading latent diffusion models.
• The method enables image generation at a rate of less than 0.3 seconds per image using a masked autoregressive model with Diffusion Loss, demonstrating high inference speed.
• The MAR variant with Diffusion Loss achieves a FID of 2.31 (without CFG) and 1.78 (with CFG), significantly outperforming previous systems like MaskGIT and MAGE.
• Replacing causal attention with bidirectional attention in the MAR framework leads to a substantial FID improvement, from 13.07 to 3.43 (without CFG), highlighting the importance of context modeling.
• The method shows strong scaling behavior, with performance improving across model sizes, suggesting potential for further gains with larger architectures.
• The approach eliminates the need for complex, gradient-approximation-sensitive vector-quantized tokenizers, simplifying training and improving reconstruction quality.