[Paper Review] CogView: Mastering Text-to-Image Generation via Transformers

Ming Ding, Zhuoyi Yang+9 more|arXiv (Cornell University)|May 26, 2021

Generative Adversarial Networks and Image Synthesis42 references383 citations

TL;DR

CogView trains a 4B-parameter Transformer with a VQ-VAE image tokenizer to perform high-quality text-to-image generation, achieving strong FID scores and enabling downstream finetuning tasks.

ABSTRACT

Text-to-Image generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding. We propose CogView, a 4-billion-parameter Transformer with VQ-VAE tokenizer to advance this problem. We also demonstrate the finetuning strategies for various downstream tasks, e.g. style learning, super-resolution, text-image ranking and fashion design, and methods to stabilize pretraining, e.g. eliminating NaN losses. CogView achieves the state-of-the-art FID on the blurred MS COCO dataset, outperforming previous GAN-based models and a recent similar work DALL-E.

Motivation & Objective

• Aim to build a scalable cross-modal model that can generate images from text in a general domain.
• Develop a discrete image tokenizer (VQ-VAE) and a large Transformer to model text and image tokens jointly.
• Propose stabilization techniques (Precision Bottleneck Relaxation and Sandwich LayerNorm) for training very large transformers.
• Demonstrate finetuning strategies for downstream tasks like style learning, super-resolution, captioning, and self-reranking.

Proposed method

• Adopt a two-stage framework where an image tokenizer (VQ-VAE) compresses images to discrete tokens and a GPT-like Transformer models the joint token sequence.
• Use 4B parameter Transformer with 48 layers, hidden size 2560, 40 attention heads, processing sequences up to 1088 tokens.
• Train on 30M Chinese text–image pairs with left-to-right token prediction for both text and image tokens.
• Stabilize training with Precision Bottleneck Relaxation (PB-relax) and Sandwich LayerNorm (Sandwich-LN) to prevent NaN/overflow and manage value scales.
• Employ three-region sparse attention to speed up training and enable larger models.
• Finetune CogView for downstream tasks such as super-resolution, image captioning/self-reranking (via CapLoss), and style learning.

Experimental results

Research questions

• RQ1 Can a large-scale transformer with a VQ-VAE image tokenizer achieve state-of-the-art text-to-image generation in a general domain?
• RQ2 What techniques are necessary to stabilize training of very large text-to-image transformers on heterogeneous data?
• RQ3 How can pretrained models be finetuned for downstream tasks like super-resolution, captioning, and style transfer without additional large-cluster resources?
• RQ4 How does self-reranking based on internal metrics compare to external CLIP-based reranking?
• RQ5 How do the generated images compare to baselines in both automated metrics (FID, IS, CapLoss) and human evaluation?

Key findings

• CogView achieves strong automated metrics on blurred MS COCO, surpassing prior GAN-based methods and closely competing with DALL-E in key settings.
• Self-reranking using Caption Loss improves image selection without needing CLIP, outpacing CLIP-based reranking in FID on MS COCO in their setup.
• Finetuning enables super-resolution, image captioning/self-reranking, and style learning, with examples including fashion design and multiple artistic styles.
• PB-relax and Sandwich-LN effectively stabilize training of a 4B (and extrapolations to larger) Transformer on a heterogeneous text–image corpus, reducing NaN occurrences.
• Compared to DALL-E, CogView demonstrates competitive FID on a standard benchmark and advantages in stability and downstream finetuning capabilities.
• The model supports domain-specific style transfer and high-resolution guidance via patch-wise super-resolution finetuning.