[Paper Review] Control Color: Multimodal Diffusion-based Interactive Image Colorization

Zhexin Liang, Zhaochen Li+3 more|arXiv (Cornell University)|Feb 16, 2024

Image Retrieval and Classification Techniques4 citations

TL;DR

This paper proposes CtrlColor, a multimodal diffusion-based image colorization framework that leverages Stable Diffusion with user-guided controls for highly controllable, interactive colorization. By integrating text prompts, user-drawn strokes, and exemplar images within a unified latent diffusion framework, it achieves state-of-the-art performance in color richness, fidelity, and user control, significantly reducing color overflow and miscoloring through content-guided deformable convolution and streamlined self-attention guidance.

ABSTRACT

Despite the existence of numerous colorization methods, several limitations still exist, such as lack of user interaction, inflexibility in local colorization, unnatural color rendering, insufficient color variation, and color overflow. To solve these issues, we introduce Control Color (CtrlColor), a multi-modal colorization method that leverages the pre-trained Stable Diffusion (SD) model, offering promising capabilities in highly controllable interactive image colorization. While several diffusion-based methods have been proposed, supporting colorization in multiple modalities remains non-trivial. In this study, we aim to tackle both unconditional and conditional image colorization (text prompts, strokes, exemplars) and address color overflow and incorrect color within a unified framework. Specifically, we present an effective way to encode user strokes to enable precise local color manipulation and employ a practical way to constrain the color distribution similar to exemplars. Apart from accepting text prompts as conditions, these designs add versatility to our approach. We also introduce a novel module based on self-attention and a content-guided deformable autoencoder to address the long-standing issues of color overflow and inaccurate coloring. Extensive comparisons show that our model outperforms state-of-the-art image colorization methods both qualitatively and quantitatively.

Motivation & Objective

• To address key limitations in image colorization, including lack of user interaction, inflexible local editing, color overflow, and unnatural color rendering.
• To unify unconditional, text-based, stroke-based, and exemplar-based colorization within a single diffusion framework for greater flexibility and consistency.
• To reduce color overflow and miscoloring in diffusion-based colorization using content-aware guidance mechanisms.
• To enable precise, iterative, and intuitive local color editing through stroke-based control and multi-modal conditioning.
• To achieve superior visual quality and user satisfaction compared to existing methods through comprehensive evaluation and user study.

Proposed method

• Integrates text prompts, user-drawn strokes, and exemplar images as multi-modal conditions in a pre-trained Stable Diffusion model for conditional image generation.
• Encodes user strokes via a hint point map and a derived binary mask to guide denoising in the latent space, enabling precise local color control.
• Introduces a content-guided deformable autoencoder in the decoder to align generated colors with input textures, reducing color overflow and improving fidelity.
• Applies streamlined self-attention guidance during inference to blur and regenerate out-of-distribution attention regions, minimizing small-scale color bleeding.
• Uses a unified framework to support all combinations of conditions (text, strokes, exemplars), enabling flexible, iterative editing.
• Leverages rich priors from the pre-trained SD model to generate diverse, realistic, and colorful outputs without fine-tuning.

Figure 2 : Visual comparisons on stroke-based colorization.

Experimental results

Research questions

• RQ1 Can a unified diffusion-based framework effectively support multiple modalities (text, strokes, exemplars) for interactive image colorization?
• RQ2 How can color overflow and miscoloring be reduced in diffusion-based colorization without fine-tuning the backbone model?
• RQ3 To what extent can user-drawn strokes enable precise, localized color control in a latent diffusion setting?
• RQ4 How does the integration of exemplar images influence color distribution consistency and realism in generated outputs?
• RQ5 Does the proposed method outperform existing state-of-the-art methods in both quantitative metrics and human evaluation?

Key findings

• CtrlColor achieves state-of-the-art performance in FID (6.8830 with 50–100 strokes), PSNR (28.2699), SSIM (0.9926), and LPIPS (0.0872), demonstrating superior fidelity and reconstruction accuracy.
• The method significantly reduces color overflow and miscoloring, as shown by visual comparisons and ablation studies, especially when using the deformable autoencoder and self-attention guidance.
• User study results show that CtrlColor outperforms DDColor, ToVivid, BigColor, and UniColor in all four evaluation categories: perceptual realism, color richness, aesthetic sense, and faithful reconstruction.
• With more strokes, the FID score decreases (from 8.8614 to 6.8830), indicating improved alignment with ground truth, though colorfulness slightly decreases due to reduced use of model priors.
• Prompt-based colorization achieves better CLIP and Colorfulness scores than UniColor, indicating stronger alignment with text descriptions and more vivid results.
• Exemplar-based colorization successfully preserves the color distribution of reference images while maintaining visual realism and consistency across objects.

Figure 3 : Left : The main structure of our CtrlColor Model achieves multi-modal controllable colorization by blending controls from diverse components. Right : To manage large color overflow and inaccurate color regions, we integrate content-guided deformable convolution layers into the autoencoder