[Paper Review] Invertible Conditional GANs for image editing

Guim Perarnau, Joost van de Weijer+2 more|arXiv (Cornell University)|Nov 19, 2016

Generative Adversarial Networks and Image Synthesis12 references584 citations

TL;DR

IcGANs combine an encoder with a conditional GAN to map real images to a latent space and edit them via conditional attributes, enabling deterministic, complex image modifications.

ABSTRACT

Generative Adversarial Networks (GANs) have recently demonstrated to successfully approximate complex data distributions. A relevant extension of this model is conditional GANs (cGANs), where the introduction of external information allows to determine specific representations of the generated images. In this work, we evaluate encoders to inverse the mapping of a cGAN, i.e., mapping a real image into a latent space and a conditional representation. This allows, for example, to reconstruct and modify real images of faces conditioning on arbitrary attributes. Additionally, we evaluate the design of cGANs. The combination of an encoder with a cGAN, which we call Invertible cGAN (IcGAN), enables to re-generate real images with deterministic complex modifications.

Motivation & Objective

• Motivate and enable complex image editing by combining inference (encoder) with conditional image generation.
• Learn to map real images into a latent z and conditional y space for controllable editing.
• Evaluate encoder designs and the placement of conditional information in the GAN architecture.
• Demonstrate editing on MNIST and CelebA datasets and analyze reconstruction quality and attribute control.

Proposed method

• Introduce IcGANs, integrating an encoder with a cGAN to obtain (z, y) = E(x) and reconstruct images x' = G(z, y).
• Train E in two parts: Ez to recover latent z from G(z, y′) outputs, and Ey to recover attributes y from real images.
• Investigate encoder designs (SNG, IND, IND-COND) and find IND as most effective.
• Explore two design decisions for cGANs: optimal placement of conditional vector y in generator and discriminator, and sampling of y′ from py.
• Evaluate cGAN conditioning using an attribute predictor (Anet) to measure how well generated images reflect conditioned attributes.
• Use CelebA and MNIST datasets to validate reconstruction quality and attribute manipulation.

Experimental results

Research questions

• RQ1 Can an encoder be learned to invert a conditional GAN and recover latent and attribute representations from real images?
• RQ2 What is the best strategy for integrating conditional information y into the generator and discriminator to maximize attribute fidelity?
• RQ3 Which encoder architecture (SNG, IND, IND-COND) yields the most accurate latent and attribute reconstructions?
• RQ4 How well do IcGANs reconstruct real images and allow controlled modification of attributes on MNIST and CelebA?
• RQ5 Does interpolating or swapping latent z and attributes y produce plausible and smooth transitions between images?

Key findings

• The IcGAN enables to map real images to latent z and attributes y, allowing reconstruction and complex attribute-based editing.
• Best cGAN conditioning position is y inserted at the input of the generator and in the first layer of the discriminator for optimal attribute fidelity.
• Two independent encoders (IND) for z and y provide the lowest reconstruction error among tested configurations.
• Attribute prediction accuracy for generated CelebA images with y conditioning reaches about 86% on mean accuracy across attributes, with higher fidelity when y is conditioned early in the network.
• IcGAN reconstructions preserve high-level features and allow meaningful attribute edits on CelebA and MNIST, with smooth interpolation and attribute transfer across real images.
• Reconstructed samples show that the model generalizes beyond training data, evidenced by successful manipulation of unseen real images.