[Paper Review] Deep Koalarization: Image Colorization using CNNs and Inception-ResNet-v2

Federico Baldassarre, Diego González Morín+1 more|arXiv (Cornell University)|Dec 9, 2017

Generative Adversarial Networks and Image Synthesis20 references84 citations

TL;DR

A CNN-based colorization model combines a trained-from-scratch encoder with high-level features from Inception-ResNet-v2 to color grayscale images in an end-to-end, fully convolutional framework and evaluates public acceptance via a user study.

ABSTRACT

We review some of the most recent approaches to colorize gray-scale images using deep learning methods. Inspired by these, we propose a model which combines a deep Convolutional Neural Network trained from scratch with high-level features extracted from the Inception-ResNet-v2 pre-trained model. Thanks to its fully convolutional architecture, our encoder-decoder model can process images of any size and aspect ratio. Other than presenting the training results, we assess the "public acceptance" of the generated images by means of a user study. Finally, we present a carousel of applications on different types of images, such as historical photographs.

Motivation & Objective

• Explore end-to-end deep learning approaches for automatic image colorization.
• Leverage high-level semantic features from a pre-trained Inception-ResNet-v2 to aid color estimation.
• Develop a fully convolutional encoder-decoder architecture that handles images of varying size and aspect ratios.
• Assess the plausibility of generated colors through a human user study.
• Demonstrate applications on historical photographs and other image types.

Proposed method

• Use CIE L*a*b* space and predict a* and b* channels from the luminance a input image.
• Extract a mid/high-level feature embedding from a pre-trained Inception-ResNet-v2 by feeding a 299x299 stacked luminance image.
• Fuse the Inception embedding with encoder features via a fusion layer that replicates the embedding across spatial locations.
• Train an encoder using eight 3x3 conv layers to produce a 512-channel, H/8 x W/8 feature map.
• Decode with up-sampling and convolutions to output a 2-channel a*b* map, combining with L* to form the color image.
• Optimize the Mean Square Error between predicted and ground-truth a*b* components using the Adam optimizer.

Experimental results

Research questions

• RQ1 Can a CNN-based encoder-decoder colorization model produce photorealistic color images when guided by high-level features from a pre-trained network?
• RQ2 Does integrating Inception-ResNet-v2 embeddings improve colorization over purely learned-from-scratch encoders?
• RQ3 How does the proposed architecture perform on images of different sizes and aspect ratios?
• RQ4 What is the perceived realism of the colorized images as measured by human evaluators?
• RQ5 Can the model color historical photographs, and what are its limitations?

Key findings

• The model can produce near-photorealistic colorizations for some images, especially with recognizable high-level features like skies, seas, and forests.
• Some results show color saturation limitations and conservative, grayish hues for certain regions.
• Public acceptance in a user study averaged a non-trivial rate of fooling observers, indicating credible colorizations for selected images.
• Comparison with other methods shows instances where the proposed approach outperforms in some cases and underperforms in others depending on content.
• The approach enables processing of images of varying sizes due to its fully convolutional design, and uses a small ImageNet subset for training.
• Historical photographs can be recolored, providing subjective assessments in the absence of ground truth.