[Paper Review] Generative Visual Manipulation on the Natural Image Manifold

Jun-Yan Zhu, Philipp Krähenbühl+2 more|arXiv (Cornell University)|Sep 12, 2016

Advanced Vision and Imaging42 references121 citations

TL;DR

This paper learns a natural image manifold with GANs and uses it as a constraint to perform real-time, photorealistic image edits on and transfer edits to real photos. It enables shape/color edits, generative transformations, and image-from-scribbles generation.

ABSTRACT

Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result. Unless the user has considerable artistic skill, it is easy to "fall off" the manifold of natural images while editing. In this paper, we propose to learn the natural image manifold directly from data using a generative adversarial neural network. We then define a class of image editing operations, and constrain their output to lie on that learned manifold at all times. The model automatically adjusts the output keeping all edits as realistic as possible. All our manipulations are expressed in terms of constrained optimization and are applied in near-real time. We evaluate our algorithm on the task of realistic photo manipulation of shape and color. The presented method can further be used for changing one image to look like the other, as well as generating novel imagery from scratch based on user's scribbles.

Motivation & Objective

• Motivate and enable user-controlled, photorealistic image editing that stays on the natural image manifold.
• Learn the image manifold from data using GANs and constrain edits to remain on this manifold.
• Provide real-time, gradient-based editing operations (color, shape, warping) and reliable edit transfer to original photos.
• Enable generation of new images from scribbles and morphing between images within the learned manifold.

Proposed method

• Train a GAN to approximate the natural image manifold as M~ = {G(z)} and use Euclidean latent distance as a perceptual similarity surrogate.
• Project real images onto the GAN manifold by finding the closest latent z via optimization or a learned encoder P(x; θP).
• Edit in latent space by solving a constrained optimization (Equation 5) that enforces user constraints while staying near z0 and on the manifold.
• Transfer edits to the original high-resolution image using a dense motion+color flow (Equation 6) and guided upsampling.
• Provide an interactive UI with brush-based coloring, sketching (HOG features), and warping constraints (Equation 5).
• Support three applications: realistic manipulation of existing photos, generative transformation between images, and interactive image generation from scribbles.

Experimental results

Research questions

• RQ1 Can a GAN-learned natural image manifold serve as a safe, controllable constraint for image editing?
• RQ2 How can user edits be expressed as constraints while ensuring outputs remain on the manifold and close to the original image?
• RQ3 What is the efficacy of projecting real photos onto the GAN manifold and back-propagating edits to the original image?
• RQ4 Can edits be transferred from generated images to real photos with high fidelity using dense correspondence techniques?
• RQ5 How do different editing constraints (coloring, sketching, warping) perform in producing realistic results?

Key findings

• The hybrid projection (learned encoder plus optimization) yields better reconstruction than optimization or encoder alone.
• Per-dataset reconstruction errors show the hybrid method consistently outperforming both optimization-based and network-based projections (Shoes/Church Outdoor/Outdoor Natural/Handbags/Shirts).
• Edit updates run at 50–100 ms, enabling near-real-time interactive feedback; final edit transfer to high resolution takes 5–10 seconds.
• User edits constrained to the GAN manifold produce more photorealistic results than vanilla GAN outputs, with shape+color edits achieving higher perceived realism in a user study.
• The method enables three capabilities: realistic image manipulation, generative transformation between images, and image generation from user scribbles.
• Motion+color flow-based edit transfer reduces artifacts when applying edits to the original image and can be upsampled with guided filtering for higher resolution results.