[Paper Review] High-Resolution Image Synthesis with Latent Diffusion Models

Robin Rombach, Andreas Blattmann+3 more|arXiv (Cornell University)|Dec 20, 2021

Generative Adversarial Networks and Image SynthesisComputer Science701 citations

TL;DR

Latent Diffusion Models (LDMs) train diffusion models in a perceptually equivalent, lower-dimensional latent space via a pretrained autoencoder, enabling high-resolution image synthesis with substantially reduced compute and flexible cross-attention conditioning for text or layouts.

ABSTRACT

By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve a new state of the art for image inpainting and highly competitive performance on various tasks, including unconditional image generation, semantic scene synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs. Code is available at https://github.com/CompVis/latent-diffusion .

Motivation & Objective

• Reduce computational demands of diffusion models by training and sampling in a latent space rather than pixel space.
• Preserve high fidelity and detail while enabling megapixel-scale synthesis through efficient autoencoding.
• Introduce cross-attention conditioned diffusion for versatile multimodal inputs (text, semantic maps, layouts).
• Demonstrate competitive or state-of-the-art results across unconditional, conditional, inpainting, and super-resolution tasks, with reusable pretrained models.

Proposed method

• Train a perceptual autoencoder to learn a low-dimensional latent space z = E(x) with controlled downsampling factor f; use a decoder D to reconstruct x from z.
• Train a latent diffusion model with the objective L_LDM = E_{E(x), ε ~ N(0,1), t}[||ε − ε_θ(z_t, t)||^2], where z_t is a noisy latent at step t.
• Augment the UNet backbone with cross-attention to condition generation on modalities y via a domain-specific encoder τ_θ(y), enabling text, semantic layouts, or other inputs.
• Implement a two-stage framework where diffusion operates in latent space, reducing computation while leveraging CNN-based 2D convolutions suitable for image data.
• Allow flexible conditioning through cross-attention-based mechanisms, enabling text-to-image, layout-to-image, and other multi-modal tasks (with classifier-free guidance to boost quality).
• Provide pretrained latent encoders/decoders and LDMs for reuse in various tasks (unconditional, text-to-image, inpainting, super-resolution).

Experimental results

Research questions

• RQ1 Can a diffusion model operating in a learned latent space match pixel-space diffusion quality while reducing training and sampling cost?
• RQ2 Does cross-attention conditioning enable robust multimodal generation (text, layouts, semantic maps) within latent diffusion?
• RQ3 How does perceptual compression (latent downsampling factor f) affect synthesis quality and efficiency across high-resolution tasks?
• RQ4 Can LDMs deliver competitive or state-of-the-art results across unconditional generation, text-to-image, inpainting, and super-resolution at high resolutions?
• RQ5 Are pretrained latent autoencoders reusable across different diffusion models and tasks?

Key findings

• Latent diffusion in a learned latent space yields strong high-resolution synthesis with substantially lower compute than pixel-space diffusion.
• Unconditional LDMs achieve state-of-the-art or competitive FID and precision/recall on CelebA-HQ, FFHQ, LSUN-Churches, LSUN-Bedrooms, and ImageNet-class conditioning, with fewer parameters and training resources.
• Cross-attention conditioning enables effective text-to-image, layout-to-image, and semantic synthesis, achieving competitive MS-COCO text-to-image results with fewer parameters than comparable AR or diffusion baselines.
• LDM-SR (super-resolution) shows strong FID and perceptual quality, often outperforming SR3 in FID while maintaining competitive perceptual metrics; user studies favor LDM-based outputs.
• Convolutional sampling in latent space enables larger-scale image generation (up to megapixel) with consistent quality and better sample throughput than pixel-space diffusion approaches.
• Overall, LDMs deliver a flexible, reusable framework that reduces computational demands without sacrificing fidelity across multiple high-resolution image synthesis tasks.