[Paper Review] Making Images Real Again: A Comprehensive Survey on Deep Image Composition

Li Niu, Wenyan Cong+5 more|ArXiv.org|Jun 28, 2021

Digital Media Forensic Detection103 references38 citations

TL;DR

A comprehensive survey of deep image composition, detailing sub-tasks (object placement, blending, harmonization, shadow generation, generative composition, foreground search), datasets, evaluation metrics, and a roadmap for future work.

ABSTRACT

As a common image editing operation, image composition (object insertion) aims to combine the foreground from one image and another background image, to produce a composite image. However, there are many issues that could make the composite images unrealistic. These issues can be summarized as the inconsistency between foreground and background, which includes appearance inconsistency (e.g., incompatible illumination), geometry inconsistency (e.g., unreasonable size), and semantic inconsistency (e.g., mismatched semantic context). The image composition task could be decomposed into multiple sub-tasks, in which each sub-task targets one or more issues. Specifically, object placement aims to find reasonable scale, location, and shape for the foreground. Image blending aims to address the unnatural boundary between foreground and background. Image harmonization aims to adjust the illumination statistics of foreground. Shadow (resp., reflection) generation aims to generate plausible shadow (resp., reflection) for the foreground. These sub-tasks can be executed sequentially or in parallel to acquire realistic composite images. To the best of our knowledge, there is no previous survey on image composition. In this paper, we conduct a comprehensive survey over the sub-tasks and combined task of image composition. For each one, we summarize the existing methods, available datasets, and common evaluation metrics. Datasets and codes for image composition are summarized at https://github.com/bcmi/Awesome-Object-Insertion. We have also contributed the first image composition toolbox: libcom https://github.com/bcmi/libcom, which assembles 10+ image-composition-related functions. The ultimate goal of this toolbox is to solve all image composition problems with simple `import libcom'. Based on libcom toolbox, we also develop an online image composition workbench https://libcom.ustcnewly.com.

Motivation & Objective

• Summarize the types of inconsistencies that degrade realism in image composition (appearance, geometry, semantic).
• Catalog sub-tasks and combinatorial tasks to achieve realistic composites (placement, blending, harmonization, shadow generation, generative composition, foreground search).
• Provide datasets, evaluation metrics, and practical guidance to researchers building and evaluating image composition models.

Proposed method

• Classify issues in image composition into appearance, geometry, and semantic inconsistencies and map them to corresponding sub-tasks.
• Describe traditional and deep learning approaches for each sub-task and how they can be combined sequentially or in parallel.
• Summarize available datasets (e.g., OPA, COCO-derived data) and evaluation metrics used in object placement and image composition research.
• Highlight the role of diffusion models and generative methods in unified or multi-task composition pipelines.
• Offer guidance on datasets, benchmarks, and evaluation practices for fair comparison across methods.

Experimental results

Research questions

• RQ1 What are the main inconsistency types in image composition and which sub-tasks address them?
• RQ2 How can object placement, blending, harmonization, and shadow generation be organized into effective pipelines (sequential or parallel) to produce realistic composites?
• RQ3 What datasets and evaluation metrics are used to assess image composition methods, and what are their strengths and limitations?
• RQ4 How do generative (including diffusion-based) methods compare to discriminative/optimization-based approaches in object placement and composition?
• RQ5 What are practical guidelines and benchmarks for future work in deep image composition?

Key findings

• Object placement, blending, harmonization, and shadow generation address appearance, boundary, and illumination inconsistencies to improve realism.
• Discriminative object placement models can outperform generative ones in locating plausible placements, especially when augmented with efficient score maps for location scoring.
• Generative approaches offer flexibility for diverse placements and view changes, but may be less efficient or require more supervision in some settings.
• Datasets like OPA provide a large-scale benchmark with rationality labels to evaluate composite realism.
• There is no unified benchmark for image blending, but existing methods utilize realism scores, user studies, PSNR against ground-truth composites, and downstream task performance as proxies.