[Paper Review] Attribute2Image: Conditional Image Generation from Visual Attributes

Xinchen Yan, Shuicheng Yan+2 more|arXiv (Cornell University)|Dec 2, 2015

Generative Adversarial Networks and Image Synthesis54 references95 citations

TL;DR

This paper proposes Attribute2Image, a conditional variational auto-encoder with a layered generative model that disentangles foreground and background latent variables to generate diverse, realistic images from visual attributes. By modeling image generation as a composition of foreground and background layers with shape-aware blending, the method achieves state-of-the-art results in attribute-conditioned image generation, reconstruction, and completion, with quantitative improvements in attribute similarity and sample quality.

ABSTRACT

This paper investigates a novel problem of generating images from visual attributes. We model the image as a composite of foreground and background and develop a layered generative model with disentangled latent variables that can be learned end-to-end using a variational auto-encoder. We experiment with natural images of faces and birds and demonstrate that the proposed models are capable of generating realistic and diverse samples with disentangled latent representations. We use a general energy minimization algorithm for posterior inference of latent variables given novel images. Therefore, the learned generative models show excellent quantitative and visual results in the tasks of attribute-conditioned image reconstruction and completion.

Motivation & Objective

• To address the challenge of generating realistic, diverse images conditioned on high-level visual attributes such as gender, age, expression, and hair color.
• To improve image generation quality by modeling images as layered compositions of foreground objects and backgrounds with disentangled latent factors.
• To enable posterior inference on novel images for tasks like image reconstruction and completion using a general optimization-based method.
• To demonstrate that disentangled latent representations lead to better attribute control and sample diversity in conditional image generation.

Proposed method

• The model uses a conditional variational auto-encoder (CVAE) with separate encoders and decoders for foreground and background latent variables, enabling end-to-end training via back-propagation and the reparameterization trick.
• Foreground and background are generated separately: the foreground is conditioned on attributes and latent code, while the background is generated from its own latent code, with shape maps controlling background visibility.
• Image composition is achieved by summing the foreground layer and a gated background layer, where the gate is determined by the foreground's shape map.
• A general energy minimization algorithm is used for posterior inference on novel images, enabling reconstruction and completion by optimizing latent variables given observed image patches.
• The model employs a disentangled CVAE (disCVAE) architecture with shared attribute streams and separate encoder/decoder networks for foreground and background, using convolutional and fully-connected layers with skip connections.
• The training objective is a variational lower bound that includes KL divergence terms for posterior approximation and reconstruction losses for image and shape map prediction.

Experimental results

Research questions

• RQ1 Can a deep generative model produce realistic and diverse images conditioned on visual attributes while maintaining disentangled representations of foreground and background factors?
• RQ2 How well can the model reconstruct or complete images from partial observations using posterior inference with learned priors?
• RQ3 Does explicit modeling of foreground and background as separate layers improve image generation quality compared to joint modeling?
• RQ4 To what extent do the disentangled latent variables in the model reflect meaningful semantic attributes such as gender, age, and expression?
• RQ5 How does the proposed optimization-based posterior inference method compare to baselines in image reconstruction and attribute accuracy?

Key findings

• The disCVAE model achieves a cosine similarity of 0.9057 in the attribute space, significantly outperforming nearest neighbor (0.8719) and blurred nearest neighbor (0.8291) baselines.
• The mean squared error (MSE) for predicted attributes on generated samples is 16.71, which is lower than the nearest neighbor baseline (21.88), indicating better attribute fidelity.
• The model generates diverse and realistic samples conditioned on attributes, with qualitative results showing high visual fidelity and accurate attribute alignment.
• Posterior inference via optimization enables effective image reconstruction and completion, demonstrating the model's robustness to partial observations.
• The disentangled latent space enables meaningful disentanglement of foreground and background factors, as shown by visual analysis of generated layers.
• The ablation study confirms that explicit foreground-background modeling improves attribute accuracy and sample quality compared to joint modeling.