[Paper Review] Learning to Detect Multiple Photographic Defects

Ning Yu, Xiaohui Shen+3 more|arXiv (Cornell University)|Dec 6, 2016

Visual Attention and Saliency Detection3 citations

TL;DR

This paper introduces a multi-task learning framework using a multi-column deep convolutional neural network (CNN) to simultaneously detect and rank the severity of seven common photographic defects—such as bad exposure, noise, blur, and composition issues—across natural images. Trained on a large-scale human-annotated dataset of 12,853 images, the model achieves higher consistency with human judgments than prior low-level statistics-based methods and even outperforms an average human user in defect severity prediction.

ABSTRACT

In this paper, we introduce the problem of simultaneously detecting multiple photographic defects. We aim at detecting the existence, severity, and potential locations of common photographic defects related to color, noise, blur and composition. The automatic detection of such defects could be used to provide users with suggestions for how to improve photos without the need to laboriously try various correction methods. Defect detection could also help users select photos of higher quality while filtering out those with severe defects in photo curation and summarization. To investigate this problem, we collected a large-scale dataset of user annotations on seven common photographic defects, which allows us to evaluate algorithms by measuring their consistency with human judgments. Our new dataset enables us to formulate the problem as a multi-task learning problem and train a multi-column deep convolutional neural network (CNN) to simultaneously predict the severity of all the defects. Unlike some existing single-defect estimation methods that rely on low-level statistics and may fail in many cases on natural photographs, our model is able to understand image contents and quality at a higher level. As a result, in our experiments, we show that our model has predictions with much higher consistency with human judgments than low-level methods as well as several baseline CNN models. Our model also performs better than an average human from our user study.

Motivation & Objective

• To address the lack of methods for simultaneously detecting multiple photographic defects in natural images.
• To collect a large-scale, human-annotated dataset of 12,853 images with severity scores for seven common defects.
• To develop a deep learning model that predicts multiple defect types in parallel, improving accuracy over single-defect methods.
• To enable applications in automated photo enhancement, curation, and quality filtering by predicting defect severity and location.
• To demonstrate that deep learning models trained on real-world human judgments outperform both traditional statistical methods and baseline CNNs.

Proposed method

• The authors collected a dataset of 12,853 natural images with human-annotated severity scores for seven photographic defects: bad exposure, bad white balance, over/under saturation, noise, haze, undesired blur, and bad composition.
• They formulated the problem as a multi-task learning task, training a multi-column CNN that processes both global image features and local image patches.
• The network architecture combines global image input with local patch inputs to balance high-level content understanding and low-level statistical features.
• The model was trained to predict continuous severity scores for each defect, with loss functions optimized for consistency with human annotations.
• For localization, the network was converted to a fully convolutional network (FCN) with upsampling to generate spatially varying defect heatmaps.
• The model was evaluated using Kendall’s rank correlation coefficient (ρ) to measure consistency with human judgments and compared against baseline methods on real and synthetic data.

Figure 1: An illustration of detecting multiple photographic defects. For each defect (from left to right: bad exposure , bad white balance , over/under saturation , noise , haze , undesired blur , bad composition ), we report the relative ranking of a severity score in percentage, compared to all t

Experimental results

Research questions

• RQ1 Can a deep learning model trained on human-annotated data achieve higher consistency with human judgments than low-level statistics-based methods in detecting multiple photographic defects?
• RQ2 Does a multi-column CNN that leverages both global and local image features outperform single-defect estimation baselines in multi-defect detection?
• RQ3 Can the proposed model generalize to synthetic defects, and how does performance compare between real and synthetic data?
• RQ4 Does the model's performance exceed that of an average human user in defect severity prediction?
• RQ5 Can the model generate spatially aware defect maps to localize defective image regions?

Key findings

• The proposed multi-column CNN model achieved a mean Kendall’s rank correlation (ρ) of 0.956 with human judgments across all seven defects, significantly outperforming low-level statistics-based methods and baseline CNNs.
• The model outperformed an average human user in defect severity prediction, as measured by the consistency of rankings against ground truth from four other annotators.
• On synthetic data, the model achieved a mean ρ of 0.9568 across five defect types, indicating strong generalization and robustness to global defects.
• The model demonstrated superior performance on challenging cases, such as differentiating motion blur from shallow depth-of-field, where prior methods failed.
• The fully convolutional adaptation enabled spatially varying defect localization, producing heatmaps that highlight regions with high defect severity, such as overexposed shadows or blurred faces.
• The dataset of 12,853 human-annotated images is publicly released to support future research in multi-defect detection and image quality assessment.

Figure 3: The histogram of the ground-truth severity scores for the bad exposure defect, shown with a coarse bin size (left) and with a fine bin size (right). Each bin in the left figure corresponds to a peak in the right figure. The hiogram has 11 peaks due to the discrete levels used in user annot