[Paper Review] UniColor: A Unified Framework for Multi-Modal Colorization with Transformer

Zhitong Huang, Nanxuan Zhao+1 more|arXiv (Cornell University)|Sep 22, 2022

Media, Gender, and Advertising4 citations

TL;DR

UniColor proposes the first unified Transformer-based framework for multi-modal image colorization, supporting unconditional, stroke, exemplar, text, and hybrid controls by converting all conditions into a common hint point representation. It achieves state-of-the-art performance in diversity, quality, and flexibility across all modalities, enabling interactive, iterative, and hybrid color editing through a two-stage architecture combining Chroma-VQGAN and Hybrid-Transformer.

ABSTRACT

We propose the first unified framework UniColor to support colorization in multiple modalities, including both unconditional and conditional ones, such as stroke, exemplar, text, and even a mix of them. Rather than learning a separate model for each type of condition, we introduce a two-stage colorization framework for incorporating various conditions into a single model. In the first stage, multi-modal conditions are converted into a common representation of hint points. Particularly, we propose a novel CLIP-based method to convert the text to hint points. In the second stage, we propose a Transformer-based network composed of Chroma-VQGAN and Hybrid-Transformer to generate diverse and high-quality colorization results conditioned on hint points. Both qualitative and quantitative comparisons demonstrate that our method outperforms state-of-the-art methods in every control modality and further enables multi-modal colorization that was not feasible before. Moreover, we design an interactive interface showing the effectiveness of our unified framework in practical usage, including automatic colorization, hybrid-control colorization, local recolorization, and iterative color editing. Our code and models are available at https://luckyhzt.github.io/unicolor.

Motivation & Objective

• Address the lack of unified frameworks that support diverse, multi-modal colorization controls (e.g., stroke, text, exemplar) in a single model.
• Overcome the limitation of existing methods that are restricted to single-modality controls and cannot generalize across modalities.
• Enable flexible, interactive colorization workflows including hybrid controls, local recolorization, and iterative editing.
• Maintain high diversity and visual quality in generated colorizations while supporting user-guided customization.
• Design a system that supports practical, real-time user interaction for legacy and modern images.

Proposed method

• Convert multi-modal inputs (stroke, exemplar, text) into a unified intermediate representation using hint points—pixels annotated with color and spatial position.
• Introduce a CLIP-based method to localize text-desired objects and generate corresponding hint points by projecting text embeddings to image regions.
• Use a two-stage framework: first, encode grayscale and conditional hint points; second, generate diverse colorizations via a Transformer-based network.
• Design Chroma-VQGAN with separate gray and color encoders and a joint decoder to disentangle content and color representation.
• Develop Hybrid-Transformer with cross-attention between hint points and latent tokens to condition color generation on multi-modal cues.
• Support iterative editing and local recolorization by masking and resampling color tokens within user-defined regions.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework effectively support diverse, multi-modal colorization controls (e.g., stroke, exemplar, text) within a single architecture?
• RQ2 How can different modalities with distinct distributions (e.g., sparse strokes vs. dense exemplars vs. semantic text) be mapped into a common, effective intermediate representation?
• RQ3 To what extent can a Transformer-based generative model maintain high diversity and visual quality in colorization while being conditioned on heterogeneous hint points?
• RQ4 Can the unified framework support practical, interactive workflows such as hybrid control, local recolorization, and iterative editing?
• RQ5 What are the failure modes in multi-modal control fusion, and how can conflicts (e.g., overlapping hint points) be detected or resolved?

Key findings

• UniColor outperforms state-of-the-art methods in both unconditional and conditional colorization across all evaluated modalities, demonstrating superior diversity and quality.
• The CLIP-based text-to-hint-point method successfully localizes objects described in text and generates semantically relevant color cues.
• The two-stage framework with Chroma-VQGAN and Hybrid-Transformer enables high-fidelity, diverse colorization results even under complex hybrid controls.
• The interactive system supports practical use cases including automatic colorization, hybrid-control editing, local recolorization, and iterative refinement.
• The framework enables end-to-end colorization with user-defined regions for recoloring, such as changing a jacket from green to orange or a bus from yellow to red.
• Failure cases include unexpected colorings (e.g., green road) and control conflicts (e.g., overlapping red and green hints), suggesting room for improvement in sampling constraints and conflict resolution.