[Paper Review] GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

Alex Nichol, Prafulla Dhariwal+6 more|arXiv (Cornell University)|Dec 20, 2021

Generative Adversarial Networks and Image Synthesis997 citations

TL;DR

GLIDE trains a 3.5B text-conditional diffusion model and compares classifier-free and CLIP-guided sampling, finding classifier-free guidance yields more photorealistic and caption-aligned images; it also demonstrates text-driven inpainting edits and releases a filtered, smaller model for safety.

ABSTRACT

Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique to trade off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples. Samples from a 3.5 billion parameter text-conditional diffusion model using classifier-free guidance are favored by human evaluators to those from DALL-E, even when the latter uses expensive CLIP reranking. Additionally, we find that our models can be fine-tuned to perform image inpainting, enabling powerful text-driven image editing. We train a smaller model on a filtered dataset and release the code and weights at https://github.com/openai/glide-text2im.

Motivation & Objective

• Advance text-conditional image synthesis with diffusion models at scale (3.5B parameters) while evaluating guidance strategies for fidelity and caption alignment.
• Demonstrate text-driven image editing and inpainting using diffusion models.
• Assess safety implications and provide a filtered release variant to mitigate misuse.

Proposed method

• Train a 3.5B parameter text-conditional diffusion model conditioned on natural language prompts.
• Compare classifier-free guidance and CLIP guidance for steering diffusion from noised images.
• Fine-tune the model for image inpainting with mask conditioning to enable text-driven edits.
• Train a smaller, filtered GLIDE (filtered) model on a cleaned dataset to reduce misuse risk.
• Train a noised CLIP model to enable CLIP-guided diffusion with noised inputs.

Experimental results

Research questions

• RQ1 Does classifier-free guidance outperform CLIP guidance for text-conditioned diffusion in terms of photorealism and caption similarity?
• RQ2 Can diffusion models be effectively steered to perform high-quality text-conditioned generation without a separate classifier?
• RQ3 How well can the model perform image inpainting/editing driven by natural language prompts?
• RQ4 What safety measures (data filtering, smaller filtered models) mitigate risks of misuse like disinformation or deepfakes?
• RQ5 How does GLIDE compare to prior text-to-image models (e.g., DALL-E) in human evaluations?

Key findings

• Classifier-free guidance yields higher-quality, more photorealistic images and better caption alignment than CLIP guidance in human evaluations.
• GLIDE samples with classifier-free guidance are preferred to DALL-E in human studies (87% photorealism, 69% caption similarity in certain settings).
• GLIDE achieves competitive MS-COCO FID (12.24) without MS-COCO-specific training, and remains strong under a filtered evaluation (12.89).
• GLIDE enables text-conditioned inpainting that matches surrounding style and lighting, including shadows and reflections.
• A smaller filtered GLIDE model is released to reduce misuse risk, with data filtering removing people and certain violent/hate content; red-teaming suggests limited ability to generate recognizable human images from filtered data.