[Paper Review] Hierarchical Text-Conditional Image Generation with CLIP Latents

Aditya Ramesh, Prafulla Dhariwal+3 more|arXiv (Cornell University)|Apr 13, 2022

Generative Adversarial Networks and Image Synthesis2,265 citations

TL;DR

The paper presents unCLIP, a two-stage text-to-image system that first generates CLIP image embeddings from captions via a prior (autoregressive or diffusion) and then decodes images from those embeddings using diffusion models, achieving higher diversity with comparable photorealism to GLIDE. It also enables image manipulation in CLIP latent space and demonstrates diffusion priors for efficiency.

ABSTRACT

Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.

Motivation & Objective

• Leverage CLIP representations to build a hierarchical text-to-image generator that separates caption-to-embedding prior from embedding-to-image decoding.
• Incorporate diffusion-based decoding and priors to improve diversity, efficiency, and high-resolution generation.
• Explore capabilities such as image variations, interpolations, and language-guided (text-diff) image manipulations in CLIP space.
• Evaluate against existing text-to-image models on diversity, realism, and caption alignment, including zero-shot MS-COCO performance.

Proposed method

• Train a decoder diffusion model that inverts the CLIP image encoder to map embeddings to images conditioned on CLIP image and (optionally) text embeddings.
• Train a prior model to generate CLIP image embeddings z_i from captions y; use autoregressive (AR) or diffusion priors with classifier-free guidance.
• For AR prior, compress z_i with PCA (319 components), quantize to 1024 buckets, and model with a Transformer predicting a token sequence conditioned on y and z_t.
• For diffusion prior, model the continuous z_i with a diffusion model conditioned on y (and optionally z_t), predicting unnoised z_i directly.
• Train two diffusion upsamplers to produce 64→256→1024 resolution images with targeted degradation and random crops to stabilize training.
• Enable guidance at sampling time to improve fidelity without collapsing diversity, via classifier-free guidance and conditioning dropout.

Experimental results

Research questions

• RQ1 Can CLIP latent space be effectively used as an intermediate representation for text-to-image generation with diffusion decoders?
• RQ2 Do diffusion priors offer better efficiency and sample quality than autoregressive priors when generating CLIP embeddings from captions?
• RQ3 What is the impact of conditioning on CLIP embeddings versus only on text prompts for photorealism, caption similarity, and diversity?
• RQ4 How does the unCLIP stack compare to GLIDE in zero-shot MS-COCO FID and human judgments across fidelity, caption alignment, and diversity?
• RQ5 What manipulation capabilities arise from CLIP latent space, such as variations, interpolations, and text-dacented edits?

Key findings

• A two-stage unCLIP model can generate diverse, photorealistic images driven by text prompts, with diffusion priors achieving higher quality and lower compute than autoregressive priors.
• Diffusion priors outperform AR priors in comparable model sizes and training compute for text-to-image generation.
• The model achieves a state-of-the-art MS-COCO 256×256 FID of 10.39 (zero-shot) when using the diffusion prior, outperforming many contemporaries.
• Human evaluations show unCLIP broadly competitive with GLIDE in photorealism and better diversity, with GLIDE sometimes edging in caption similarity.
• Guidance during decoding increases aesthetic quality for both GLIDE and unCLIP, while unCLIP maintains diversity better under guided sampling compared to GLIDE.