[论文解读] CCC++: Optimized Color Classified Colorization with Segment Anything Model (SAM) Empowered Object Selective Color Harmonization

Mrityunjoy Gain, Avi Deb Raha+1 more|arXiv (Cornell University)|Mar 18, 2024

Color perception and design被引用 7

一句话总结

CCC++ 将上色（colorization）重新表述为一个颜色分类问题，具备特征平衡，并引入基于 SAM 的对象选择性色彩协调，达到更好的饱和度和边缘质量。

ABSTRACT

In this paper, we formulate the colorization problem into a multinomial classification problem and then apply a weighted function to classes. We propose a set of formulas to transform color values into color classes and vice versa. To optimize the classes, we experiment with different bin sizes for color class transformation. Observing class appearance, standard deviation, and model parameters on various extremely large-scale real-time images in practice we propose 532 color classes for our classification task. During training, we propose a class-weighted function based on true class appearance in each batch to ensure proper saturation of individual objects. We adjust the weights of the major classes, which are more frequently observed, by lowering them, while escalating the weights of the minor classes, which are less commonly observed. In our class re-weight formula, we propose a hyper-parameter for finding the optimal trade-off between the major and minor appeared classes. As we apply regularization to enhance the stability of the minor class, occasional minor noise may appear at the object's edges. We propose a novel object-selective color harmonization method empowered by the Segment Anything Model (SAM) to refine and enhance these edges. We propose two new color image evaluation metrics, the Color Class Activation Ratio (CCAR), and the True Activation Ratio (TAR), to quantify the richness of color components. We compare our proposed model with state-of-the-art models using six different dataset: Place, ADE, Celeba, COCO, Oxford 102 Flower, and ImageNet, in qualitative and quantitative approaches. The experimental results show that our proposed model outstrips other models in visualization, CNR and in our proposed CCAR and TAR measurement criteria while maintaining satisfactory performance in regression (MSE, PSNR), similarity (SSIM, LPIPS, UIUI), and generative criteria (FID).

研究动机与目标

• 解决在自动着色中由于背景特征占主导，前景对象被低估的问题引发的特征不平衡。
• 将连续的颜色值转换为离散的颜色类别，以实现更平衡的学习和更丰富的颜色分布。
• 引入类别加权损失和超参数调优，以平衡主要颜色类别和次要颜色类别。
• 开发一个基于 SAM 的对象选择性颜色协调模块，以优化对象边缘和颜色边界。
• 提出新的颜色图像评估指标（CCAR 和 TAR），以量化颜色丰富度和激活精度。

提出的方法

• 将双通道 a*b* 颜色分量转换为单通道的 532 个离散颜色类别集合；通过实验分箱大小来确定最佳颗粒度。
• 使用基于每个批次中真实类别出现的加权交叉熵损失，以平衡主要颜色类别和次要颜色类别。
• 应用数据驱动的、超参数调优的类别重加权方案，以防止常见类别支配，同时保护少数类别。
• 利用 Segment Anything Model (SAM) 进行对象选择性颜色协调，以细化对象边缘和颜色一致性。
• 引入 Color Class Activation Ratio (CCAR) 和 True Activation Ratio (TAR) 作为新的颜色图像评估指标。
• 在六个数据集（Place、ADE、CelebA、COCO、Oxford 102 Flower、ImageNet）上进行评估，并使用多种回归、相似性和生成指标进行比较。

实验结果

研究问题

• RQ1 将颜色值转换为离散颜色类别是否能缓解特征不平衡并提高小物体的颜色饱和度？
• RQ2 带有批次感知权重的类别加权损失对着色图像的颜色分布和边缘质量有何影响？
• RQ3 基于 SAM 的对象选择性颜色协调是否提高对象边界处的边缘细化和颜色保真度？
• RQ4 CCAR 和 TAR 是否为评估着色输出中的颜色丰富度和激活度提供有意义的指标？

主要发现

• 识别出 532 个颜色类别作为着色任务的最优选择。
• 基于批次的类别加权方案平衡主要和次要颜色类别，以减少去饱和和过饱和偏差。
• 基于 SAM 的对象选择性颜色协调细化对象边缘并改善颜色边界。
• CCC++ 在可视化质量以及 CCAR/TAR 指标上超过了最先进的基线，同时在回归（MSE、PSNR）、相似性（SSIM、LPIPS、UIQI）和生成（FID）等分数方面保持竞争力。
• 该方法在六个大规模数据集上得到验证（Place、ADE、CelebA、COCO、Oxford 102 Flower、ImageNet）。
• 引入 CCAR 和 TAR 作为量化着色输出中色彩丰富度和激活精度的新指标。