[Paper Review] DDColor: Towards Photo-Realistic Image Colorization via Dual Decoders

Xiaoyang Kang, Tao Yang+4 more|arXiv (Cornell University)|Dec 22, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

DDColor proposes an end-to-end image colorization method using dual decoders: a pixel decoder for spatial reconstruction and a query-based transformer decoder that learns semantic-aware color queries from multi-scale features without hand-crafted priors. This design significantly reduces color bleeding and enhances color richness, achieving state-of-the-art performance on ImageNet, COCO-Stuff, and ADE20k with superior qualitative and quantitative results.

ABSTRACT

Image colorization is a challenging problem due to multi-modal uncertainty and high ill-posedness. Directly training a deep neural network usually leads to incorrect semantic colors and low color richness. While transformer-based methods can deliver better results, they often rely on manually designed priors, suffer from poor generalization ability, and introduce color bleeding effects. To address these issues, we propose DDColor, an end-to-end method with dual decoders for image colorization. Our approach includes a pixel decoder and a query-based color decoder. The former restores the spatial resolution of the image, while the latter utilizes rich visual features to refine color queries, thus avoiding hand-crafted priors. Our two decoders work together to establish correlations between color and multi-scale semantic representations via cross-attention, significantly alleviating the color bleeding effect. Additionally, a simple yet effective colorfulness loss is introduced to enhance the color richness. Extensive experiments demonstrate that DDColor achieves superior performance to existing state-of-the-art works both quantitatively and qualitatively. The codes and models are publicly available at https://github.com/piddnad/DDColor.

Motivation & Objective

• To address the ill-posed nature of image colorization, particularly multi-modal uncertainty and semantic inconsistency in deep learning-based methods.
• To overcome limitations of prior transformer-based methods that rely on manually designed priors, suffer from poor generalization, or exhibit color bleeding in complex scenes.
• To improve color richness and semantic consistency in automatic image colorization without user guidance or external priors.
• To develop an end-to-end framework that jointly optimizes spatial detail and semantic-aware color representation.

Proposed method

• The method employs a dual-decoder architecture: a pixel decoder restores high-resolution spatial features, while a query-based color decoder learns color embeddings from multi-scale visual features using cross-attention.
• The color decoder uses learnable queries to attend to multi-scale image features, enabling semantic-aware color representation learning without hand-crafted priors.
• Cross-attention mechanisms establish dynamic correlations between semantic features and color queries, reducing color bleeding and improving context-aware colorization.
• A novel colorfulness loss is introduced to explicitly enhance the vibrancy and richness of generated color outputs.
• The final colorization result is produced by fusing outputs from both decoders, with the final image formed by concatenating the predicted AB channels with the input L channel.
• The model is trained end-to-end using perceptual and colorfulness losses, with ablation studies validating component importance.

Experimental results

Research questions

• RQ1 Can a dual-decoder architecture with learnable color queries improve semantic consistency and reduce color bleeding in image colorization?
• RQ2 How does the use of multi-scale features in the color decoder affect performance on complex scenes with small or diverse objects?
• RQ3 To what extent can a learnable, end-to-end color decoder replace manually designed priors in image colorization?
• RQ4 Does a dedicated colorfulness loss significantly improve the visual richness of generated colorizations?
• RQ5 How well does the proposed method generalize to datasets beyond ImageNet, such as COCO-Stuff and ADE20k, without fine-tuning?

Key findings

• DDColor achieves the best FID score of 3.92 on ImageNet, outperforming previous state-of-the-art methods, including those using classification-based color space discretization.
• The model with 100 color queries achieves optimal performance, reducing FID to 3.92 and increasing colorfulness (CF) to 38.26, with minimal degradation in ΔCF.
• Ablation studies confirm that both cross-attention and self-attention layers in the color decoder are essential, with performance dropping significantly when either is removed.
• The full model with multi-scale features eliminates visible color bleeding and improves edge accuracy, while single-scale variants show significant artifacts on object boundaries.
• On COCO-Stuff and ADE20k, DDColor achieves the best performance among all baselines without fine-tuning, demonstrating strong generalization ability.
• Visualization of color queries shows that each query specializes in specific semantic regions (e.g., fur, skin, grass), confirming their semantic relevance and localized color learning.