[Paper Review] Image Generation and Editing with Variational Info Generative AdversarialNetworks

Mahesh Gorijala, Ambedkar Dukkipati|arXiv (Cornell University)|Jan 17, 2017

Generative Adversarial Networks and Image Synthesis4 references8 citations

TL;DR

This paper proposes Variational InfoGAN (ViGAN), a generative model that combines Variational Autoencoders (VAEs) and InfoGAN to jointly model images and their visual descriptions. By integrating an inference network with disentangled, interpretable latent codes, ViGAN enables conditional image generation and attribute-based image editing while producing sharper samples than VAEs and more disentangled representations than standard GANs.

ABSTRACT

Recently there has been an enormous interest in generative models for images in deep learning. In pursuit of this, Generative Adversarial Networks (GAN) and Variational Auto-Encoder (VAE) have surfaced as two most prominent and popular models. While VAEs tend to produce excellent reconstructions but blurry samples, GANs generate sharp but slightly distorted images. In this paper we propose a new model called Variational InfoGAN (ViGAN). Our aim is two fold: (i) To generated new images conditioned on visual descriptions, and (ii) modify the image, by fixing the latent representation of image and varying the visual description. We evaluate our model on Labeled Faces in the Wild (LFW), celebA and a modified version of MNIST datasets and demonstrate the ability of our model to generate new images as well as to modify a given image by changing attributes.

Motivation & Objective

• To develop a generative model that jointly learns image and visual description representations for improved disentanglement and interpretability.
• To enable conditional image generation by conditioning on visual descriptions or attributes.
• To allow editing of existing images by modifying only specific attributes while preserving other content.
• To combine the reconstruction strength of VAEs with the sample quality of GANs, overcoming limitations of both.
• To jointly train an inference network with the generative model, enabling end-to-end learning of posterior inference and generation.

Proposed method

• Introduces a hybrid architecture combining VAEs and InfoGAN, using a subset of latent variables (c) to represent disentangled, interpretable attributes.
• Employs an encoder network to infer the posterior distribution q(z|x) for image input x, enabling end-to-end training with the generator and discriminator.
• Uses a mutual information objective between the disentangled latent code c and the generated image x to encourage meaningful attribute disentanglement.
• Trains the model via a joint loss function combining VAE reconstruction loss, KL divergence, GAN adversarial loss, and attribute-specific recognition loss.
• Applies a multi-head recognition head to predict visual attributes from the latent code, enabling attribute editing during inference.
• Uses Adam optimizer with separate learning rates for generator, discriminator, encoder, and recognizer, with hyperparameters λ₁ and λ₂ balancing reconstruction and attribute sensitivity.

Experimental results

Research questions

• RQ1 Can a generative model jointly learn disentangled representations of image attributes and generate realistic, conditional images?
• RQ2 Can the model perform meaningful image editing by modifying only specific attributes while preserving other content?
• RQ3 Does combining VAE and InfoGAN improve sample quality and disentanglement compared to standalone VAEs or GANs?
• RQ4 Can end-to-end training of the inference network with the generator and discriminator yield better posterior approximation and generation performance?
• RQ5 How sensitive is the model's editing capability to hyperparameter choices, particularly λ₁ and λ₂?

Key findings

• ViGAN successfully generates high-quality, sharp images that are more realistic than VAE-generated samples, which tend to be blurry.
• The model demonstrates effective attribute-based image editing: changing digit labels in MNIST, adding eyeglasses in CelebA, and modifying facial expressions in LFW while preserving spatial structure.
• Reconstructions on MNIST, CelebA, and LFW show that the model maintains high fidelity, with the encoder accurately inferring latent codes from input images.
• The model achieves better disentanglement of attributes than standard GANs, as evidenced by controlled edits where only the intended attribute changes.
• The performance of image editing is highly sensitive to hyperparameters λ₁ and λ₂, requiring careful tuning to balance reconstruction accuracy and attribute sensitivity.
• Qualitative comparisons show that samples from ViGAN are more realistic than those from VAEs and GANs alone, indicating a successful fusion of strengths from both models.