[Paper Review] Learning a Discriminative Model for the Perception of Realism in Composite Images

Jun-Yan Zhu, Philipp Krähenbühl+2 more|arXiv (Cornell University)|Oct 2, 2015

Advanced Vision and Imaging33 references14 citations

TL;DR

This paper proposes a CNN-based discriminative model that learns to predict visual realism in composite images by distinguishing natural photos from automatically generated composites, using only unlabeled data. The model captures realism through color, lighting, and texture compatibility, and outperforms prior methods in human perception studies by guiding automatic color adjustment and object selection for compositing.

ABSTRACT

What makes an image appear realistic? In this work, we are answering this question from a data-driven perspective by learning the perception of visual realism directly from large amounts of data. In particular, we train a Convolutional Neural Network (CNN) model that distinguishes natural photographs from automatically generated composite images. The model learns to predict visual realism of a scene in terms of color, lighting and texture compatibility, without any human annotations pertaining to it. Our model outperforms previous works that rely on hand-crafted heuristics, for the task of classifying realistic vs. unrealistic photos. Furthermore, we apply our learned model to compute optimal parameters of a compositing method, to maximize the visual realism score predicted by our CNN model. We demonstrate its advantage against existing methods via a human perception study.

Motivation & Objective

• To develop a data-driven method for assessing visual realism in composite images without human annotations.
• To learn visual realism cues—such as color, lighting, and texture compatibility—directly from large-scale unlabeled composites.
• To apply the learned model to optimize compositing parameters, improving realism automatically.
• To evaluate the model’s effectiveness via human perception studies comparing it to existing methods.

Proposed method

• Train a deep CNN to classify natural images versus automatically generated composites using a large dataset of object-swapped composites.
• Use the trained model as a realism score predictor for new images, ranking composites by perceived realism.
• Optimize color mapping functions by maximizing the predicted realism score, with regularization to avoid unrealistic color shifts.
• Apply hard negative mining by retraining the CNN with newly generated, poorly performing composites to improve robustness.
• Use the model to select the most realistic foreground object from a database by ranking candidate composites.
• Evaluate results via pairwise human comparisons using Thurstone's Case V Model for reliability.

Experimental results

Research questions

• RQ1 Can a CNN trained on unlabeled composites learn to predict human perception of visual realism?
• RQ2 What visual cues (e.g., color, lighting, texture) does the model learn to associate with realism?
• RQ3 Can the learned realism score guide automatic color adjustment to produce more realistic composites?
• RQ4 Does the model improve object selection for compositing compared to shape-based or random selection?
• RQ5 How does the model’s performance compare to prior state-of-the-art compositing methods in human perception studies?

Key findings

• The model achieved a human rating of 0.311 for compositing improvements, significantly outperforming [15] (-0.247) and [33] (-0.237).
• After three iterations of hard negative mining, the model's realism score improved from -0.162 to 0.117, reducing extreme color adjustments.
• The RealismCNN method achieved a human rating of 0.285 for object selection, outperforming shape-based selection (-0.033) and random selection (-0.252).
• The model successfully ranked composites by perceived realism, with images near the decision boundary judged as realistic by most human subjects.
• The model primarily captures low-level realism cues such as color, lighting, and texture compatibility, though it does not model high-level scene semantics or layout.
• The method demonstrated generalization by improving compositing results across diverse image categories and real-world backgrounds.