[Paper Review] GP-GAN: Towards Realistic High-Resolution Image Blending

Huikai Wu, Shuai Zheng+2 more|arXiv (Cornell University)|Mar 21, 2017

Advanced Image Processing Techniques44 references89 citations

TL;DR

GP-GAN combines a Blending GAN for low-resolution color realism with a Gaussian-Poisson optimization to produce high-resolution, artifact-free image blends. It achieves state-of-the-art realism on the Transient Attributes dataset and is supported by user studies.

ABSTRACT

It is common but challenging to address high-resolution image blending in the automatic photo editing application. In this paper, we would like to focus on solving the problem of high-resolution image blending, where the composite images are provided. We propose a framework called Gaussian-Poisson Generative Adversarial Network (GP-GAN) to leverage the strengths of the classical gradient-based approach and Generative Adversarial Networks. To the best of our knowledge, it's the first work that explores the capability of GANs in high-resolution image blending task. Concretely, we propose Gaussian-Poisson Equation to formulate the high-resolution image blending problem, which is a joint optimization constrained by the gradient and color information. Inspired by the prior works, we obtain gradient information via applying gradient filters. To generate the color information, we propose a Blending GAN to learn the mapping between the composite images and the well-blended ones. Compared to the alternative methods, our approach can deliver high-resolution, realistic images with fewer bleedings and unpleasant artifacts. Experiments confirm that our approach achieves the state-of-the-art performance on Transient Attributes dataset. A user study on Amazon Mechanical Turk finds that the majority of workers are in favor of the proposed method.

Motivation & Objective

• Motivate realistic high-resolution image blending for composite images.
• Leverage GANs to learn a low-resolution color-realism constraint.
• Combine gradient-based high-frequency detail with color constraints via a Gaussian-Poisson framework.
• Demonstrate state-of-the-art performance on a challenging dataset and via user studies.

Proposed method

• Introduce Blending GAN to generate a low-resolution realistic blended image from a composite input.
• Train Blending GAN with a supervised loss combining L2 and adversarial loss.
• Formulate Gaussian-Poisson Equation to fuse color information from the low-res constraint with gradient information from the high-res composite, solved in closed form over a Laplacian pyramid.
• Apply a multi-scale (Laplacian pyramid) procedure to produce a high-resolution blended image by iteratively solving at each scale.
• Provide a practical algorithm (Algorithm 1) that generates the final high-resolution blend by integrating the Gaussion-Poisson optimization with pyramid upsampling.

Experimental results

Research questions

• RQ1 Can GANs learn a plausible low-resolution color blending that, when combined with gradient-based cues, yield realistic high-resolution blends?
• RQ2 Does a Gaussian-Poisson formulation provide high-quality high-frequency detail while preserving color consistency with a learned low-resolution constraint?
• RQ3 How does GP-GAN perform against classical Poisson-based blending and other image stitching methods on realism and artifacts?
• RQ4 Is the approach robust across masks and real-world high-resolution images beyond the training distribution?

Key findings

• GP-GAN outperforms Poisson-based and multi-spline baselines in realism scores on 500 images from the Transient Attributes dataset.
• User study on Amazon Mechanical Turk shows GP-GAN is preferred by the majority of subjects for realism.
• Replacing the Blending GAN color constraint with a simple down-sampled composite reduces quality, highlighting the importance of the learned low-resolution realism constraint.
• Qualitative results show GP-GAN produces high-resolution blends with fewer bleedings and illumination inconsistencies compared to MPB and MSB.
• The method generalizes to real high-resolution images and does not require precise object masks for good results.