[Paper Review] CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers

Ming Ding, Wendi Zheng+2 more|arXiv (Cornell University)|Apr 28, 2022

Multimodal Machine Learning ApplicationsComputer Science121 citations

TL;DR

CogView2 uses hierarchical transformers with local parallel autoregressive generation and a cross-modal general language model (CogLM) to speed up high-resolution text-to-image generation, achieving competitive results to DALL-E-2 and enabling interactive editing.

ABSTRACT

The development of the transformer-based text-to-image models are impeded by its slow generation and complexity for high-resolution images. In this work, we put forward a solution based on hierarchical transformers and local parallel auto-regressive generation. We pretrain a 6B-parameter transformer with a simple and flexible self-supervised task, Cross-modal general language model (CogLM), and finetune it for fast super-resolution. The new text-to-image system, CogView2, shows very competitive generation compared to concurrent state-of-the-art DALL-E-2, and naturally supports interactive text-guided editing on images.

Motivation & Objective

• Motivate faster, scalable text-to-image generation at higher resolutions.
• Develop a unified pretraining framework (CogLM) that handles text and image tokens bidirectionally.
• Enable hierarchical generation to separate low- and high-resolution processing for efficiency.
• Introduce local parallel autoregressive refinement (LoPAR) to speed up high-resolution image synthesis.

Proposed method

• Propose CogLM: a 6B-parameter cross-modal transformer pretraining with text and image tokens and mask-based autoregressive prediction.
• Use two mask-based pretraining tasks: full image token masking (text-to-image) and mixed image patches with text masking (infilling and captioning).
• Adopt a hierarchical generation pipeline: 1) generate 20x20 token low-res images, 2) map to 60x60 tokens via direct super-resolution, 3) refine with iterative LoPAR-based super-resolution.
• Implement 2D local attention with a custom CUDA kernel to accelerate local computations during super-resolution and refinement.
• Improve sampling and training efficiency via cluster-based sampling of image tokens and a textual attention upweighting technique to boost text-image relevance.

Experimental results

Research questions

• RQ1 Can a cross-modal, bidirectional pretraining objective improve generation and downstream tasks for text-to-image models?
• RQ2 Does a hierarchical generation pipeline coupled with local attention and LoPAR enable fast, high-resolution image synthesis without sacrificing quality?
• RQ3 Can targeted attention mechanisms and sampling strategies improve the fidelity and relevance of generated images to the input text?

Key findings

• CogView2 achieves competitive Fréchet Inception Distance (FID) and Inception Score (IS) compared to state-of-the-art, with FID-0 around 24.0 and IS around 22.4 on MS-COCO without COCO finetuning.
• Finetuning CogLM on MS-COCO improves FID to 17.5 (FID-0) and reduces FID across other scales (e.g., FID-4 to 10.6, FID-8 to 10.4), though human evaluation may favor non-finetuned models for certain scenes.
• Hierarchical generation with LoPAR substantially speeds up high-resolution generation (up to 600x faster than CogView-style autoregressive generation) while maintaining or improving image quality.
• Cluster sampling for token generation mitigates incomplete truncation in large vocabularies, improving sampling coherence.
• Local CUDA-based attention accelerates 2D local attention, achieving up to 40x speedups over full attention in autoregressive scenarios.