[Paper Review] Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space

Anh‐Tu Nguyen, Jeff Clune+3 more|arXiv (Cornell University)|Nov 30, 2016

Generative Adversarial Networks and Image Synthesis17 references158 citations

TL;DR

This paper introduces Plug-and-Play Generative Networks (PPGN), a method that enhances image generation by combining a pretrained generator with a conditional classifier and a learned latent prior, enabling high-resolution (227×227), diverse, and photo-realistic image synthesis across all 1000 ImageNet classes through iterative sampling in latent space. The approach achieves state-of-the-art quality and diversity by unifying activation maximization under a probabilistic energy-based framework with Metropolis-adjusted Langevin dynamics.

ABSTRACT

Generating high-resolution, photo-realistic images has been a long-standing goal in machine learning. Recently, Nguyen et al. (2016) showed one interesting way to synthesize novel images by performing gradient ascent in the latent space of a generator network to maximize the activations of one or multiple neurons in a separate classifier network. In this paper we extend this method by introducing an additional prior on the latent code, improving both sample quality and sample diversity, leading to a state-of-the-art generative model that produces high quality images at higher resolutions (227x227) than previous generative models, and does so for all 1000 ImageNet categories. In addition, we provide a unified probabilistic interpretation of related activation maximization methods and call the general class of models "Plug and Play Generative Networks". PPGNs are composed of 1) a generator network G that is capable of drawing a wide range of image types and 2) a replaceable "condition" network C that tells the generator what to draw. We demonstrate the generation of images conditioned on a class (when C is an ImageNet or MIT Places classification network) and also conditioned on a caption (when C is an image captioning network). Our method also improves the state of the art of Multifaceted Feature Visualization, which generates the set of synthetic inputs that activate a neuron in order to better understand how deep neural networks operate. Finally, we show that our model performs reasonably well at the task of image inpainting. While image models are used in this paper, the approach is modality-agnostic and can be applied to many types of data.

Motivation & Objective

• To address the lack of diversity and quality in high-resolution image generation using activation maximization methods.
• To improve sample diversity and realism in latent space optimization by introducing a learned prior on the latent code.
• To unify existing activation maximization techniques under a probabilistic energy-based framework for generative modeling.
• To enable conditional image generation not only on ImageNet classes but also on image captions via modular 'plug-and-play' condition networks.
• To demonstrate the method's utility in multifaceted feature visualization and image inpainting.

Proposed method

• Formalizes activation maximization as an energy-based model with two components: a prior term p(h) on the latent code h and a condition term based on classifier activations.
• Uses a generator network G to map latent codes h to images, where G is pretrained and fixed during optimization.
• Applies an approximate Metropolis-adjusted Langevin sampling algorithm to iteratively sample from the energy-based model, improving mixing and diversity.
• Introduces a joint prior p(h) that encourages realistic image generation by learning from real image features in latent space.
• Enables conditional generation by optimizing h to maximize the activation of a target neuron in a classifier network C, which can be an ImageNet or captioning network.
• Supports image inpainting by conditioning on both class labels and a partial real image context, constraining h to match observed image regions.

Experimental results

Research questions

• RQ1 Can a latent space optimization method with a learned prior generate high-resolution, diverse, and photo-realistic images across all 1000 ImageNet classes?
• RQ2 How does incorporating a learned prior improve sample diversity and quality compared to standard activation maximization?
• RQ3 Can the same generator be used to generate images conditioned on different types of conditions, such as ImageNet classes or natural language captions?
• RQ4 To what extent does the PPGN framework improve multifaceted feature visualization compared to prior methods?
• RQ5 Can PPGN perform effective image inpainting by combining class conditioning and spatial context constraints?

Key findings

• PPGN generates high-resolution (227×227) images that are nearly photo-realistic and diverse across all 1000 ImageNet classes, outperforming prior methods in both quality and diversity.
• The introduction of a learned prior on the latent code significantly improves sample diversity, reducing mode collapse and enabling exploration of multiple image compositions within a class.
• The method achieves state-of-the-art performance in multifaceted feature visualization, revealing diverse and semantically meaningful stimuli that activate individual neurons.
• PPGN demonstrates strong performance in image inpainting by reconstructing missing image regions using both class labels and spatial context, producing semantically coherent results.
• Ablation studies show that the optimal balance between the prior and condition terms (e.g., ϵ1 = 10⁻⁵) maximizes image quality and diversity, while too strong a prior leads to mode collapse.
• The model generates images with higher diversity than real ImageNet training images in some cases, as quantified by lower similarity in feature space, indicating effective exploration of the data manifold.