[Paper Review] LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation

Jianwei Yang, Anitha Kannan+2 more|arXiv (Cornell University)|Mar 5, 2017

Generative Adversarial Networks and Image Synthesis137 citations

TL;DR

LR-GAN generates images by recursively composing foreground objects with separately modeled appearance, shape, and pose over a generated background, yielding more natural and recognizable images than DCGAN. It introduces foreground-background layering and spatial transformations within a GAN framework.

ABSTRACT

We present LR-GAN: an adversarial image generation model which takes scene structure and context into account. Unlike previous generative adversarial networks (GANs), the proposed GAN learns to generate image background and foregrounds separately and recursively, and stitch the foregrounds on the background in a contextually relevant manner to produce a complete natural image. For each foreground, the model learns to generate its appearance, shape and pose. The whole model is unsupervised, and is trained in an end-to-end manner with gradient descent methods. The experiments demonstrate that LR-GAN can generate more natural images with objects that are more human recognizable than DCGAN.

Motivation & Objective

• Motivate generation of natural images by leveraging the layered structure of scenes (background and foreground objects).
• Propose a recursive GAN that builds images in stages, pasting foreground layers onto a generated background.
• Decompose each object into appearance, shape (mask), and pose (affine transformation) for flexible scene composition.
• Train the model end-to-end in an unsupervised manner and demonstrate improvements over DCGAN across multiple datasets.

Proposed method

• Introduce a background generator G_b and a recurrent foreground generator G_f that share parameters across timesteps.
• At each timestep t, generate an object's appearance f_t, shape m_t, and pose a_t, transform them via a spatial transformer ST, and compose with the previous canvas x_{t-1} using Eq. (4).
• Use a mask m_t with a sigmoid output to obtain alpha-blended foregrounds and a spatial transformer grid to apply affine transformations to both f_t and m_t.
• Incorporate temporal connections via a noise-LSTM and a past-object pooling mechanism to condition new objects on prior content.
• Train with a GAN objective using a discriminator D to distinguish real from generated images, enabling end-to-end gradient-based optimization.
• Propose evaluation metrics including Adversarial Accuracy and Adversarial Divergence in addition to Inception Score.

Experimental results

Research questions

• RQ1 Can a layered recursive GAN generate more natural, recognizable images by explicitly modeling background and multiple foreground objects?
• RQ2 Does decomposing objects into appearance, shape, and pose and applying affine transformations improve foreground-background separation and scene realism?
• RQ3 How do explicit spatial transformations and masks affect the quality and contextual relevance of generated images across datasets?
• RQ4 Are the proposed metrics (Adversarial Accuracy and Adversarial Divergence) effective in assessing distributional similarity between real and generated images?
• RQ5 How does LR-GAN compare to DCGAN on datasets like MNIST variants, CIFAR-10, and CUB-200 in terms of visual fidelity and human judgments?

Key findings

• LR-GAN produces images with clearer foreground-background boundaries and less blending artifacts than DCGAN on CIFAR-10 and CUB-200.
• Qualitative and human studies show LR-GAN yields more realistic and recognizable objects, e.g., sharper bird shapes on CUB-200.
• On CIFAR-10, LR-GAN outperforms DCGAN across Inception Score variants, Adversarial Accuracy, and Adversarial Divergence metrics in the reported experiments.
• Ablation studies demonstrate the importance of affine transformations and the mask (shape) generator for avoiding degenerate decompositions and preserving plausible results.
• Contextual generation results show foregrounds that are compatible with fixed backgrounds, indicating learned contextual dependencies between layers.
• Category-specific generators improve realism for particular classes (e.g., horse, frog, cat) in CIFAR-10.