[Paper Review] ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systems

Denis Zavadski, Johann-Friedrich Feiden+1 more|arXiv (Cornell University)|Dec 11, 2023

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

ControlNet-XS rethinks text-to-image diffusion control by decoupling the control network from generative capabilities, eliminating feedback delay inherent in prior methods like ControlNet. This allows a much smaller, faster network (1.7M–55M parameters) to achieve superior image quality and control fidelity, outperforming larger models while being ~2x faster in training and inference.

ABSTRACT

The field of image synthesis has made tremendous strides forward in the last years. Besides defining the desired output image with text-prompts, an intuitive approach is to additionally use spatial guidance in form of an image, such as a depth map. In state-of-the-art approaches, this guidance is realized by a separate controlling model that controls a pre-trained image generation network, such as a latent diffusion model. Understanding this process from a control system perspective shows that it forms a feedback-control system, where the control module receives a feedback signal from the generation process and sends a corrective signal back. When analysing existing systems, we observe that the feedback signals are timely sparse and have a small number of bits. As a consequence, there can be long delays between newly generated features and the respective corrective signals for these features. It is known that this delay is the most unwanted aspect of any control system. In this work, we take an existing controlling network (ControlNet) and change the communication between the controlling network and the generation process to be of high-frequency and with large-bandwidth. By doing so, we are able to considerably improve the quality of the generated images, as well as the fidelity of the control. Also, the controlling network needs noticeably fewer parameters and hence is about twice as fast during inference and training time. Another benefit of small-sized models is that they help to democratise our field and are likely easier to understand. We call our proposed network ControlNet-XS. When comparing with the state-of-the-art approaches, we outperform them for pixel-level guidance, such as depth, canny-edges, and semantic segmentation, and are on a par for loose keypoint-guidance of human poses. All code and pre-trained models will be made publicly available.

Motivation & Objective

• Address the fundamental limitation in existing control networks: delayed information flow between generation and control processes.
• Reduce the parameter count of control networks without sacrificing performance, enabling faster training and inference.
• Improve control fidelity and image quality by focusing the control network solely on guiding the generation process.
• Mitigate semantic bias induced by large control networks that distort generation under conflicting control signals.
• Demonstrate that a minimal control network can outperform larger, more complex counterparts when feedback delay is eliminated.

Proposed method

• Decouple the control network from the generative model's architecture, removing the need for the control network to learn generative capabilities.
• Design a lightweight control network (ControlNet-XS) that processes guidance inputs (e.g., depth maps, edge maps) and injects control signals directly into the diffusion model's denoising blocks.
• Eliminate feedback delay by ensuring real-time, bidirectional information flow between the generative and control networks during training and inference.
• Use a lightweight, non-generative architecture for the control network, with only 1.7M to 55M parameters, compared to ControlNet’s ~100M.
• Inject control signals via learnable, adaptive weighting (α) at intermediate denoising blocks, preserving the generative model’s original capabilities.
• Train the control network end-to-end while freezing the generative model, using zero-convolution initialization to stabilize training.

(a) ControlNet [ 72 ]

Experimental results

Research questions

• RQ1 Can a control network be designed to avoid learning generative capabilities, thereby improving efficiency and control fidelity?
• RQ2 Does eliminating feedback delay between the generative and control networks lead to better performance with smaller models?
• RQ3 Can a significantly smaller control network outperform larger, more complex models like ControlNet in image quality and control accuracy?
• RQ4 To what extent does a large control network introduce semantic bias that distorts generation under conflicting control signals?
• RQ5 How does model size in the control network affect control fidelity and generalization under challenging, non-aligned control inputs?

Key findings

• ControlNet-XS with 1.7M parameters achieves higher image quality and control fidelity than ControlNet, despite being drastically smaller.
• The 55M-parameter ControlNet-XS model is ~2x faster than ControlNet during both training and inference, while maintaining or improving performance.
• ControlNet-XS produces superior results under conflicting control signals—e.g., generating a cake in a street scene—where ControlNet fails even after adjusting the control strength α.
• The smallest ControlNet-XS model (1.7M parameters) shows reduced control fidelity compared to larger variants, but still maintains strong guidance, indicating a trade-off between size and precision.
• ControlNet-XS mitigates semantic bias: unlike ControlNet, which imposes semantic meaning on depth maps even at low α, ControlNet-XS preserves the generative model’s original semantics.
• Quantitative evaluation on Stable Diffusion XL shows that ControlNet-XS outperforms T2I-Adapter and matches or exceeds ControlNet’s performance with a fraction of the parameters.

(b) ControlNet-XS (Ours)