[Paper Review] CCC: Color Classified Colorization

Mrityunjoy Gain, Avi Deb Raha+1 more|arXiv (Cornell University)|Mar 3, 2024

Pigment Synthesis and Properties6 citations

TL;DR

This paper reframes image colorization as a weighted color-classification task with 215 color classes, introduces SAM-based edge-aware color harmonization, and proposes a Chromatic Number Ratio (CNR) metric; it outperforms baselines across multiple datasets.

ABSTRACT

Automatic colorization of gray images with objects of different colors and sizes is challenging due to inter- and intra-object color variation and the small area of the main objects due to extensive backgrounds. The learning process often favors dominant features, resulting in a biased model. In this paper, we formulate the colorization problem into a multinomial classification problem and then apply a weighted function to classes. We propose a set of formulas to transform color values into color classes and vice versa. Class optimization and balancing feature distribution are the keys for good performance. Observing class appearance on various extremely large-scale real-time images in practice, we propose 215 color classes for our colorization task. During training, we propose a class-weighted function based on true class appearance in each batch to ensure proper color saturation of individual objects. We establish a trade-off between major and minor classes to provide orthodox class prediction by eliminating major classes' dominance over minor classes. As we apply regularization to enhance the stability of the minor class, occasional minor noise may appear at the object's edges. We propose a novel object-selective color harmonization method empowered by the SAM to refine and enhance these edges. We propose a new color image evaluation metric, the Chromatic Number Ratio (CNR), to quantify the richness of color components. We compare our proposed model with state-of-the-art models using five different datasets: ADE, Celeba, COCO, Oxford 102 Flower, and ImageNet, in both qualitative and quantitative approaches. The experimental results show that our proposed model outstrips other models in visualization and CNR measurement criteria while maintaining satisfactory performance in regression (MSE, PSNR), similarity (SSIM, LPIPS, UIQI), and generative criteria (FID).

Motivation & Objective

• Address the color imbalance problem in automatic colorization by treating color prediction as a weighted classification task.
• Optimize color-class granularity and class weights in a data-driven way to preserve minority colors.
• Integrate SAM-based object-selective color harmonization to refine object edges.
• Introduce a new Chromatic Number Ratio (CNR) metric to quantify color richness in generated images.
• Demonstrate superior performance over state-of-the-art colorization methods on multiple datasets.

Proposed method

• Transform continuous a* and b* color channels into discrete color classes using a grid-based binning formula.
• Use an encoder-decoder architecture with DenseNet encoder and CNN decoder to predict a HxW x C color-class distribution.
• Train with a cross-entropy loss and a batch-wise, data-driven class-weighting scheme to balance major/minor color classes.
• Introduce a trade-off-based class-weight adjustment (Loss_CBCE) to prevent over- or under-representation of rare colors.
• Incorporate SAM-powered object-selective color harmonization to refine edges by aligning color modes within segments.
• Propose Chromatic Number Ratio (CNR) to quantify color-class diversity in generated images.

Experimental results

Research questions

• RQ1 How can colorization be reformulated to mitigate feature (color) imbalance and preserve minority colors?
• RQ2 What data-driven methods can optimize color-class levels and class weights for robust color prediction?
• RQ3 Can SAM-based edge refinement improve color harmonization at object boundaries?
• RQ4 Does the proposed Chromatic Number Ratio (CNR) effectively measure color diversity in colorized outputs?
• RQ5 How does CCC perform relative to state-of-the-art methods across diverse datasets?

Key findings

• CCC outperforms baselines in Chromatic Number Ratio (CNR) across datasets (ADE, Celeba, COCO, Oxford 102 Flower, ImageNet).
• The 215-color-class configuration captures minority colors while maintaining overall color fidelity.
• Batch-wise class-weighting and the Loss_CBCE formulation improve color saturation of minority classes and reduce desaturation/oversaturation biases.
• SAM-empowered object-selective color harmonization sharpens edges and reduces color bleeding at boundaries.
• Quantitative results show competitive or superior performance on standard metrics (MSE, PSNR, SSIM, LPIPS, UIQI, FID) while achieving higher CNR; CCC particularly excels on the Oxford Flowers dataset.
• The proposed CNR metric provides a holistic measure of color richness beyond traditional regression/similarity metrics.