[Paper Review] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Chitwan Saharia, William Chan+12 more|arXiv (Cornell University)|May 23, 2022

Multimodal Machine Learning Applications2,103 citations

TL;DR

Imagen uses frozen large language model encoders with diffusion models to achieve photorealistic text-to-image generation and strong image-text alignment, outperforming prior methods on COCO and DrawBench. It introduces dynamic thresholding and DrawBench for evaluation.

ABSTRACT

We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment. See https://imagen.research.google/ for an overview of the results.

Motivation & Objective

• Demonstrate that freezing and scaling large text encoders improves text-to-image fidelity and alignment more than scaling diffusion models alone.
• Show that large frozen language models pretrained on text-only data can serve as effective text encoders for image synthesis.
• Introduce sampling and architecture techniques to enable high-guidance-weight diffusion with photorealism.
• Propose DrawBench as a comprehensive benchmark for evaluating text-to-image models beyond COCO.
• Assess ethical considerations and societal impact of large-scale text-to-image models and data usage.

Proposed method

• Use a frozen text encoder (pretrained T5 variants, BERT, CLIP) to map input text to embeddings while keeping encoder weights fixed.
• Employ a cascaded diffusion pipeline: a base 64x64 model conditioned on text embeddings, followed by two super-resolution diffusion models (64x64→256x256 and 256x256→1024x1024).
• Apply classifier-free guidance with high guide weights, coupled with dynamic thresholding to avoid saturation and improve fidelity.
• Introduce Efficient U-Net for memory efficiency and faster convergence in diffusion models.
• Incorporate noise conditioning augmentation for super-resolution models to improve robustness to artifacts and enable stronger text conditioning.
• Evaluate using COCO FID-30K and CLIP-alignment metrics, plus large-scale human evaluations and the DrawBench benchmark.

Experimental results

Research questions

• RQ1 Does scaling the size of frozen text encoders pretrained on text-only data yield greater gains in image fidelity and image-text alignment than scaling the diffusion model size?
• RQ2 Can high guidance weights be effectively used with diffusion models using dynamic thresholding to improve photorealism without sacrificing fidelity?
• RQ3 How do large pretrained language models compare to multi-modal text encoders (e.g., CLIP) in driving text-to-image synthesis?
• RQ4 What is the impact of architectural choices (Efficient U-Net, cross-attention text conditioning) on diffusion-based text-to-image generation?
• RQ5 How does a comprehensive benchmark like DrawBench reveal strengths and weaknesses of text-to-image models beyond COCO?

Key findings

• Scaling the text encoder size yields consistent improvements in both image-text alignment and image fidelity; T5-XXL achieves the best results.
• Text encoder scale has a larger impact than increasing the diffusion model size.
• Dynamic thresholding with large guidance weights produces significantly better photorealism and alignment than static or no thresholding.
• Noise conditioning augmentation in super-resolution stages is critical for strong text conditioning and high-quality upsampling.
• Imagen achieves a state-of-the-art zero-shot COCO FID of 7.27, outperforming GLIDE and DALL-E 2 in the reported results, and human raters match COCO captions in alignment.
• DrawBench evaluations show Imagen outperforms competing methods in both fidelity and image-text alignment, with human raters preferring Imagen across categories.
• Using large frozen language models as text encoders is effective for text-to-image generation, and cross-attention conditioning outperforms simpler pooling methods.