[Paper Review] Detecting Colorized Images via Convolutional Neural Networks: Toward High Accuracy and Good Generalization

Weize Quan, Dong‐Ming Yan+3 more|arXiv (Cornell University)|Feb 17, 2019

Digital Media Forensic Detection47 references4 citations

TL;DR

This paper proposes a deep learning-based method using a two-branch convolutional neural network (CNN) to detect colorized images with high accuracy and strong generalization, especially under blind detection conditions. By integrating a novel feature analysis branch and automatically generated negative samples via linear interpolation, the model achieves superior performance on unseen colorization algorithms while maintaining high classification accuracy.

ABSTRACT

Image colorization achieves more and more realistic results with the increasing computation power of recent deep learning techniques. It becomes more difficult to identify the fake colorized images by human eyes. In this work, we propose a novel forensic method to distinguish between natural images (NIs) and colorized images (CIs) based on convolutional neural network (CNN). Our method is able to achieve high classification accuracy and cope with the challenging scenario of blind detection, i.e., no training sample is available from "unknown" colorization algorithm that we may encounter during the testing phase. This blind detection performance can be regarded as a generalization performance. First, we design and implement a base network, which can attain better performance in terms of classification accuracy and generalization (in most cases) compared with state-of-the-art methods. Furthermore, we design a new branch, which analyzes smaller regions of extracted features, and insert it into the above base network. Consequently, our network can not only improve the classification accuracy, but also enhance the generalization in the vast majority of cases. To further improve the performance of blind detection, we propose to automatically construct negative samples through linear interpolation of paired natural and colorized images. Then, we progressively insert these negative samples into the original training dataset and continue to train the network. Experimental results demonstrate that our method can achieve stable and high generalization performance when tested against different state-of-the-art colorization algorithms.

Motivation & Objective

• To address the growing challenge of distinguishing between natural images (NIs) and colorized images (CIs) due to advanced deep learning-based colorization techniques.
• To improve detection performance in blind scenarios where test-time colorization methods are unknown during training.
• To enhance generalization capability of CNN-based forensic models without sacrificing classification accuracy.
• To develop an effective, automatic method for generating negative training samples to improve robustness against unseen colorization algorithms.

Proposed method

• Design a base end-to-end CNN model for NI/CI classification, with ablation studies on first-layer activation functions.
• Introduce a new branch that analyzes localized feature regions to capture subtle artifacts like color bleeding and monotonous coloration.
• Construct negative samples via linear interpolation between paired natural and colorized images to simulate realistic distribution gaps.
• Progressively insert these synthetic negative samples into the training set to refine the model’s generalization capacity.
• Train the network in an end-to-end manner using cross-entropy loss with standard optimization techniques.
• Visualize feature maps and convolutional kernels to interpret model behavior and validate detection cues such as color bleeding and unnatural color distribution.

Experimental results

Research questions

• RQ1 Can a CNN-based approach outperform traditional handcrafted feature methods in detecting colorized images?
• RQ2 How can a deep learning model maintain high generalization performance when tested on unseen colorization algorithms?
• RQ3 To what extent can automatically generated negative samples improve the robustness of a forensic classifier?
• RQ4 What visual cues do the model learn to distinguish between natural and colorized images?

Key findings

• The proposed two-branch CNN achieves higher classification accuracy and better generalization than state-of-the-art methods on both known and unknown colorization algorithms.
• The addition of the new feature analysis branch significantly improves the model’s ability to detect subtle artifacts such as color bleeding and unnatural color uniformity.
• The use of linearly interpolated negative samples leads to a stable and measurable improvement in generalization performance with minimal drop in classification accuracy.
• Visualization results confirm that the model learns to respond strongly to regions with color bleeding and monotonous coloration—key indicators of artificial colorization.
• Misclassification cases are primarily due to NIs with low saturation or CIs with rich, plausible coloration, indicating limitations in detecting subtle color artifacts.