[Paper Review] Adding Conditional Control to Text-to-Image Diffusion Models

Lvmin Zhang, Rao, Anyi+1 more|arXiv (Cornell University)|Feb 10, 2023

Advanced Neuroimaging Techniques and Applications204 citations

TL;DR

ControlNet adds trainable conditional controls to large pretrained text-to-image diffusion models by coupling a frozen backbone with a trainable copy via zero-initialized convolutions, enabling diverse conditioning (edges, depth, pose, etc.) with minimal data and computation.

ABSTRACT

We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, eg, edges, depth, segmentation, human pose, etc, with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.

Motivation & Objective

• Motivate finer-grained spatial control in text-to-image diffusion beyond prompts.
• Propose an end-to-end finetuning approach that preserves the backbone by freezing the pretrained model.
• Demonstrate diverse conditioning inputs (edges, depth, pose, segmentation, etc.) and assess robustness across dataset sizes.
• Show that ControlNet can compose multiple conditioning signals and operate with or without prompts.

Proposed method

• Introduce ControlNet: a trainable copy of a pretrained diffusion backbone connected to the frozen backbone via zero-initialized 1x1 convolutions.
• Attach a conditioning encoder that converts input conditioning images (e.g., edges, depth, pose) into feature maps aligned with the diffusion model’s blocks.
• Train only the ControlNet components while keeping the original model frozen to avoid catastrophic forgetting.
• Use a tiny encoder E to map conditioning images to 64x64 feature space matching the latent diffusion inputs.
• Apply the method to Stable Diffusion, inserting ControlNet blocks into encoder and middle blocks and enabling multiple conditioning signals to be composed.
• Leverage Classifier-Free Guidance (CFG) with CFG Resolution Weighting to modulate conditioning influence during inference.

Experimental results

Research questions

• RQ1 Can ControlNet enable reliable conditioning for large pretrained diffusion models without updating the backbone?
• RQ2 What conditioning modalities (edges, depth, pose, segmentation, etc.) are effective for guiding generation, and how do multiple conditions interact?
• RQ3 How does ControlNet perform with varying training data sizes and on different model variants (e.g., single vs. multiple conditions)?
• RQ4 How should CFG be adjusted to balance conditioning fidelity and output quality when using ControlNet?

Key findings

• ControlNet can steer Stable Diffusion with single or multiple conditioning inputs (edges, depth, pose, segmentation, etc.) both with and without prompts.
• Training is robust across small (<50k) and large (>1M) conditioning datasets and benefits from freezing the backbone and using zero-initialized connections.
• Zero convolutions prevent harmful noise early in training, preserving the large pretrained backbone.
• CFG Resolution Weighting allows nuanced control of conditioning influence during inference, improving fidelity without sacrificing quality.
• User studies and quantitative metrics show ControlNet outperforms several baselines in result quality and condition fidelity (e.g., higher AUR for ControlNet vs. Sketch-Guided variants).
• ControlNets can be transferred to community diffusion models without retraining the neural blocks.