[Paper Review] Zero-Shot Text-to-Image Generation

Aditya Ramesh, Mikhail Pavlov+6 more|arXiv (Cornell University)|Feb 24, 2021

Multimodal Machine Learning Applications55 references1,132 citations

TL;DR

A 12B parameter autoregressive transformer trained on 250M image-text pairs learns to generate high-fidelity images from text in a zero-shot manner, without training on paired captions. It also demonstrates rudimentary image-to-image translation and compositional capabilities.

ABSTRACT

Text-to-image generation has traditionally focused on finding better modeling assumptions for training on a fixed dataset. These assumptions might involve complex architectures, auxiliary losses, or side information such as object part labels or segmentation masks supplied during training. We describe a simple approach for this task based on a transformer that autoregressively models the text and image tokens as a single stream of data. With sufficient data and scale, our approach is competitive with previous domain-specific models when evaluated in a zero-shot fashion.

Motivation & Objective

• Demonstrate zero-shot text-to-image generation with a large-scale autoregressive transformer.
• Investigate a two-stage training pipeline combining discrete latent image tokens with text tokens.
• Evaluate zero-shot performance on MS-COCO and CUB and analyze emergent capabilities from scaling.

Proposed method

• Train a discrete VAE (dVAE) to compress 256x256 images into 32x32 image tokens (8192 codebook values).
• Train a 12B parameter sparse transformer to model the joint distribution of text and image tokens as a single stream.
• Use a two-stage ELBO objective: stage 1 optimizes phi/theta for the VAE; stage 2 optimizes psi for the prior over text+image tokens.
• Concatenate 256 BPE text tokens with 32x32 image tokens and autoregressively model them with a decoder-only transformer.
• Rerank generated samples with a pretrained contrastive model to select top images for evaluation.

Experimental results

Research questions

• RQ1 Can scale in data, model size, and training procedure enable high-quality zero-shot text-to-image generation?
• RQ2 What are the emergent capabilities (e.g., image-to-image translation, text rendering) of a large-scale model trained without caption supervision?
• RQ3 How does zero-shot performance compare to prior domain-specific models on MS-COCO and CUB?
• RQ4 What is the impact of data overlap with training sets on evaluation metrics like FID and IS?
• RQ5 What techniques are necessary to train and deploy such a large model efficiently (mixed precision, distributed optimization, gradient compression)?

Key findings

• The 12B-parameter model trained on 250M image-text pairs achieves competitive zero-shot image generation on MS-COCO without using training captions.
• Human evaluation shows the model’s samples are preferred over prior methods in realism (90%) and caption matching (93%).
• The model attains an MS-COCO FID within about 2 points of the best prior approach despite no caption supervision.
• The approach demonstrates rudimentary image-to-image translation and text rendering capabilities in zero-shot settings.
• Reranking with a contrastive model improves sample quality as the number of candidates increases (up to diminishing returns at higher N).
• There is a notable performance gap on the CUB dataset, suggesting limitations on specialized distributions without fine-tuning.