[Paper Review] Pixel-level Semantics Guided Image Colorization

Jiaojiao Zhao, Li Liu+3 more|arXiv (Cornell University)|Aug 5, 2018

Generative Adversarial Networks and Image Synthesis6 citations

TL;DR

This paper proposes a pixel-level semantics-guided image colorization method using a hierarchical deep neural network that jointly optimizes semantic segmentation and colorization losses, significantly reducing context confusion and edge color bleeding. By incorporating fine-grained object semantics and employing a joint bilateral upsampling layer, the model produces more realistic, detailed, and natural colorized images than state-of-the-art approaches.

ABSTRACT

While many image colorization algorithms have recently shown the capability of producing plausible color versions from gray-scale photographs, they still suffer from the problems of context confusion and edge color bleeding. To address context confusion, we propose to incorporate the pixel-level object semantics to guide the image colorization. The rationale is that human beings perceive and distinguish colors based on the object's semantic categories. We propose a hierarchical neural network with two branches. One branch learns what the object is while the other branch learns the object's colors. The network jointly optimizes a semantic segmentation loss and a colorization loss. To attack edge color bleeding we generate more continuous color maps with sharp edges by adopting a joint bilateral upsamping layer at inference. Our network is trained on PASCAL VOC2012 and COCO-stuff with semantic segmentation labels and it produces more realistic and finer results compared to the colorization state-of-the-art.

Motivation & Objective

• To address context confusion in automatic image colorization by leveraging pixel-level object semantics for more accurate color assignment.
• To mitigate edge color bleeding caused by scale variation during inference in deep learning-based colorization.
• To improve colorization realism by integrating translation-variant representations from semantic segmentation into the colorization process.
• To achieve superior performance on benchmark datasets through joint optimization of segmentation and colorization objectives.
• To produce more natural, saturated, and edge-preserving colorized images compared to existing methods.

Proposed method

• A dual-branch hierarchical neural network is designed: one branch performs pixel-level semantic segmentation, and the other predicts color distributions for each pixel.
• The network is jointly trained with two loss functions: a semantic segmentation loss and a colorization loss, enabling end-to-end optimization.
• A joint bilateral upsampling layer is introduced at inference to generate smoother, sharper color maps while preserving edge details.
• The model is pre-trained on PASCAL VOC2012 and COCO-Stuff with dense semantic annotations to learn rich object-specific color priors.
• The architecture leverages deep convolutional features with deconvolutional layers to recover high-resolution color outputs.
• Semantic supervision from pixel-level labels enhances the model’s ability to distinguish object categories and assign semantically plausible colors.

Experimental results

Research questions

• RQ1 Can pixel-level semantic understanding reduce context confusion in image colorization?
• RQ2 How can edge color bleeding be minimized across varying input scales in deep learning-based colorization?
• RQ3 Can joint optimization of semantic segmentation and colorization improve the realism and detail of generated color images?
• RQ4 To what extent does incorporating fine-grained semantic priors outperform image-level or global semantic priors in colorization?
• RQ5 How does the proposed joint bilateral upsampling layer compare to standard transposed convolutional upsampling in preserving color boundaries?

Key findings

• The proposed method achieves a naturalness score of 94.89% on the combined PASCAL VOC2012 and COCO-Stuff dataset, significantly outperforming the state-of-the-art methods.
• The saturability score of 95.70% indicates that the generated colors are more vivid and visually appealing than those from competing approaches.
• Semantic correctness reached 94.10%, demonstrating that the model assigns colors consistent with object categories, reducing semantic confusion.
• Edge keeping performance of 94.80% confirms the effectiveness of the joint bilateral upsampling layer in preserving sharp color transitions.
• Human evaluation showed that the proposed method is closer to ground-truth images in terms of realism, color saturation, and structural fidelity.
• Failure cases revealed limitations in rare object categories (e.g., peacock, jewelry) and small object confusion, suggesting the need for finer semantic segmentation classes.