[Paper Review] Controllable Generation with Text-to-Image Diffusion Models: A Survey

Pu Cao, Feng Zhou+2 more|arXiv (Cornell University)|Mar 7, 2024

Multimedia Communication and Technology4 citations

TL;DR

This survey provides a comprehensive analysis of controllable text-to-image diffusion models, systematically categorizing methods that extend beyond text conditioning to include diverse inputs like images, layouts, and sketches. It details theoretical mechanisms such as conditional score prediction and condition-guided score estimation, offering a unified framework for understanding how novel conditions steer image generation with high precision and flexibility.

ABSTRACT

In the rapidly advancing realm of visual generation, diffusion models have revolutionized the landscape, marking a significant shift in capabilities with their impressive text-guided generative functions. However, relying solely on text for conditioning these models does not fully cater to the varied and complex requirements of different applications and scenarios. Acknowledging this shortfall, a variety of studies aim to control pre-trained text-to-image (T2I) models to support novel conditions. In this survey, we undertake a thorough review of the literature on controllable generation with T2I diffusion models, covering both the theoretical foundations and practical advancements in this domain. Our review begins with a brief introduction to the basics of denoising diffusion probabilistic models (DDPMs) and widely used T2I diffusion models. We then reveal the controlling mechanisms of diffusion models, theoretically analyzing how novel conditions are introduced into the denoising process for conditional generation. Additionally, we offer a detailed overview of research in this area, organizing it into distinct categories from the condition perspective: generation with specific conditions, generation with multiple conditions, and universal controllable generation. For an exhaustive list of the controllable generation literature surveyed, please refer to our curated repository at https://github.com/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models.

Motivation & Objective

• To address the limitation of text-only conditioning in text-to-image diffusion models, which fails to capture complex visual requirements like specific styles or unseen identities.
• To provide a structured taxonomy of controllable generation methods based on the type of conditioning input, including single, multiple, and universal control conditions.
• To analyze the theoretical foundations of condition integration into diffusion models, focusing on conditional score prediction and condition-guided score estimation.
• To survey practical applications across image editing, inpainting, composition, and text/image-to-3D generation, highlighting real-world utility and technical innovation.
• To compile and curate a comprehensive, up-to-date repository of controllable T2I diffusion literature for researchers and practitioners.

Proposed method

• Categorizes controllable generation methods by condition type: single-condition (e.g., text, image, layout), multi-condition (e.g., text + sketch), and universal control (e.g., unified condition heads).
• Theorizes the integration of novel conditions via conditional score prediction, where the model estimates the score of a denoising step conditioned on auxiliary inputs.
• Analyzes condition-guided score estimation, which modifies the denoising process using guidance mechanisms like classifier-free guidance to align generation with complex conditions.
• Reviews architectural adaptations such as adapter modules, condition injection into U-Net, and the use of specialized heads for different control signals (e.g., ControlNet for spatial control).
• Examines the application of Score Distillation Sampling (SDS) loss in enabling text-to-3D generation by transferring 2D diffusion control mechanisms to 3D space.
• Evaluates techniques like DreamInpainter, Realfill, and Uni-inpaint for image completion, using reference images and multiple conditions to guide masked region generation.

Experimental results

Research questions

• RQ1 How can text-to-image diffusion models be extended beyond text conditioning to support diverse visual conditions such as sketches, layouts, and reference images?
• RQ2 What are the core theoretical mechanisms—such as conditional score prediction and condition-guided score estimation—that enable effective integration of novel conditions into the denoising process?
• RQ3 What are the key architectural and training strategies that allow pre-trained T2I models to be adapted for multi-condition or universal controllable generation?
• RQ4 How do controllable generation methods enhance downstream tasks like image editing, inpainting, composition, and text/image-to-3D generation?
• RQ5 What are the most effective and generalizable approaches for transferring control from 2D diffusion models to 3D generation using techniques like SDS?

Key findings

• The integration of diverse conditions—beyond text—such as reference images, sketches, and layouts significantly enhances the precision and flexibility of image generation.
• Condition-guided score estimation, particularly via classifier-free guidance, enables robust and controllable generation without requiring additional classifier training.
• Methods like ControlNet and Uni-inpaint demonstrate that multi-condition control (e.g., text + layout + sketch) leads to more coherent and semantically aligned image completions and edits.
• The use of Score Distillation Sampling (SDS) loss enables the transfer of 2D diffusion control mechanisms to 3D generation, making text-to-3D generation feasible with minimal 3D supervision.
• ObjectStitch and ControlCom show that adapter-based fine-tuning of pre-trained T2I models can unify complex composition tasks like lighting, shadow, and viewpoint alignment.
• The survey's curated repository at https://github.com/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models provides a comprehensive, up-to-date reference for the state of the art in controllable T2I generation.