[Paper Review] Controlling generative models with continuous factors of variations

Antoine Plumerault, Hervé Le Borgne+1 more|arXiv (Cornell University)|Jan 28, 2020

Generative Adversarial Networks and Image SynthesisComputer Science43 references79 citations

TL;DR

The paper introduces a method to discover interpretable latent-space directions that encode continuous image variations (like position and scale) without labels, enabling precise control of generation in GANs and VAEs.

ABSTRACT

Recent deep generative models are able to provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing. Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation. To overcome these major issues, very recent work has shown the interest of studying the semantics of the latent space of generative models. In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like the position or scale of the object in the image. Our method does not require human annotations and is particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations. We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.

Motivation & Objective

• Motivate the need for controllable generative models and interpretable latent representations.
• Propose a method to find latent-space directions corresponding to continuous factors of variation without requiring labels or encoders.
• Show that moving along latent-space directions can precisely control image properties such as object position and scale.
• Provide a reconstruction loss and optimization strategy to invert generators and estimate latent trajectories.
• Investigate how disentanglement impacts controllability and reveal latent-space structure across models.

Proposed method

• Define G:Z->I and seek z_T such that G(z_T) ≈ T_T(I) for a continuous transformation T with parameter t.
• Use a reconstruction loss L to minimize L(G(z), T(I)) with a norm constraint on z (||z|| ≤ sqrt(d)).
• Propose a texture-preserving loss L(I1,I2) = ||F{I1−I2}F{σ}||^2 to improve inversion sharpness.
• Decompose T_T into small steps and sequentially optimize z_n with previous z as initialization to traverse the latent trajectory (Algorithm 1).
• Encode the factor of variation as t = f(z) = g(<z,u>) with ||u||=1, train g_theta to predict delayed parameter changes δt from latent projections, enabling a parametric model of the factor.
• Handle unknown t by training to predict δt (not t) and learn a mapping that captures the factor distribution via equation (6) and the proposed g_theta.
• Optionally sample z to shape distributions of generated outputs using the learned g_theta and a chosen target distribution.

Experimental results

Research questions

• RQ1 Can continuous factors of variation (like position and scale) be captured as latent-space directions without labels or encoders?
• RQ2 How accurately can we control translation and scaling of objects in generated images by moving along these latent directions?
• RQ3 What reconstruction loss aids in effectively inverting generators and preserving texture during optimization?
• RQ4 How does disentanglement of the latent space affect the ability to control generation?
• RQ5 Are the identified directions shared across object categories and models (e.g., across BigGAN categories or VAEs)?

Key findings

• Latent-space directions corresponding to horizontal/vertical translation and scale enable precise control of generated images in BigGAN and β-VAE setups.
• Directions for spatial factors are largely encoded in the early parts of the BigGAN latent code; vertical position shows stronger involvement from higher-level blocks, likely due to background correlations.
• Disentangled representations (higher β in β-VAEs) improve controllability, reducing the standard deviation of the controlled factor.
• A new reconstruction loss that discounts high-frequency components yields sharper inversions and more realistic reconstructions than pixel-wise MSE.
• The proposed trajectory-based optimization does not require encoder-based inversion or additional training and can be applied directly to existing generators.
• Common directions for factors of variation appear to be shared across multiple object categories, suggesting category-agnostic latent structures.