[Paper Review] CCC++: Optimized Color Classified Colorization with Segment Anything Model (SAM) Empowered Object Selective Color Harmonization

Mrityunjoy Gain, Avi Deb Raha+1 more|arXiv (Cornell University)|Mar 18, 2024

Color perception and design7 citations

TL;DR

CCC++ reframes colorization as a color-classification problem with feature-balancing and introduces SAM-powered object-selective color harmonization, achieving improved saturation and edge quality.

ABSTRACT

In this paper, we formulate the colorization problem into a multinomial classification problem and then apply a weighted function to classes. We propose a set of formulas to transform color values into color classes and vice versa. To optimize the classes, we experiment with different bin sizes for color class transformation. Observing class appearance, standard deviation, and model parameters on various extremely large-scale real-time images in practice we propose 532 color classes for our classification task. During training, we propose a class-weighted function based on true class appearance in each batch to ensure proper saturation of individual objects. We adjust the weights of the major classes, which are more frequently observed, by lowering them, while escalating the weights of the minor classes, which are less commonly observed. In our class re-weight formula, we propose a hyper-parameter for finding the optimal trade-off between the major and minor appeared classes. As we apply regularization to enhance the stability of the minor class, occasional minor noise may appear at the object's edges. We propose a novel object-selective color harmonization method empowered by the Segment Anything Model (SAM) to refine and enhance these edges. We propose two new color image evaluation metrics, the Color Class Activation Ratio (CCAR), and the True Activation Ratio (TAR), to quantify the richness of color components. We compare our proposed model with state-of-the-art models using six different dataset: Place, ADE, Celeba, COCO, Oxford 102 Flower, and ImageNet, in qualitative and quantitative approaches. The experimental results show that our proposed model outstrips other models in visualization, CNR and in our proposed CCAR and TAR measurement criteria while maintaining satisfactory performance in regression (MSE, PSNR), similarity (SSIM, LPIPS, UIUI), and generative criteria (FID).

Motivation & Objective

• Address feature imbalance in automatic colorization where foreground objects are underrepresented due to dominant background features.
• Transform continuous color values into discrete color classes to enable balanced learning and richer color distribution.
• Introduce a class-weighted loss and hyperparameter tuning to balance major and minor color classes.
• Develop an SAM-powered object-selective color harmonization module to refine object edges and color boundaries.
• Propose new color image evaluation metrics (CCAR and TAR) to quantify color richness and activation accuracy.

Proposed method

• Transform double-channel a*b* color components into a single-channel set of 532 discrete color classes; experiment with bin sizes to determine optimal granularity.
• Use a weighted cross-entropy loss based on true class appearance in each batch to balance major and minor color classes.
• Apply a data-driven, hyper-parameter-tuned class re-weighting scheme to prevent domination by frequent classes while safeguarding minority classes.
• Leverage the Segment Anything Model (SAM) for object-selective color harmonization to refine object edges and color consistency.
• Introduce Color Class Activation Ratio (CCAR) and True Activation Ratio (TAR) as new color-image evaluation metrics.
• Evaluate on six datasets (Place, ADE, CelebA, COCO, Oxford 102 Flower, ImageNet) and compare using multiple regression, similarity, and generative metrics.

Experimental results

Research questions

• RQ1 Can transforming color values into discrete color classes mitigate feature imbalance and improve color saturation of smaller objects?
• RQ2 How does a class-weighted loss with batch-aware weighting impact color distribution and edge quality in colorized images?
• RQ3 Does SAM-powered object-selective color harmonization enhance edge refinement and color fidelity at object boundaries?
• RQ4 Do CCAR and TAR provide meaningful metrics for assessing color richness and activation in colored outputs?

Key findings

• 532 color classes were identified as optimal for the colorization task.
• A batch-driven class-weighting scheme balances major and minor color classes to reduce desaturation and over-saturation biases.
• The SAM-based object-selective color harmonization refines object edges and improves color boundaries.
• CCC++ outperforms state-of-the-art baselines in visualization quality and in CCAR/TAR metrics while maintaining competitive regression (MSE, PSNR), similarity (SSIM, LPIPS, UIQI), and generative (FID) scores.
• The approach was validated across six large-scale datasets (Place, ADE, CelebA, COCO, Oxford 102 Flower, ImageNet).
• Introduction of CCAR and TAR as novel metrics to quantify color richness and activation accuracy in colorized outputs.