[Paper Review] Semantic Interpolation in Implicit Models

Yannic Kilcher, Aurélien Lucchi+1 more|arXiv (Cornell University)|Oct 31, 2017

Generative Adversarial Networks and Image Synthesis14 references3 citations

TL;DR

This paper proposes modifying the prior distribution in implicit generative models—specifically GANs and VAEs—from a standard normal to a gamma distribution to concentrate probability mass closer to the origin. By doing so, linear interpolation in the latent space remains within high-density regions, avoiding low-probability areas that degrade sample quality. Experiments show significantly improved visual quality and more meaningful semantic interpolation in generated images across multiple datasets and latent dimensions.

ABSTRACT

In implicit models, one often interpolates between sampled points in latent space. As we show in this paper, care needs to be taken to match-up the distributional assumptions on code vectors with the geometry of the interpolating paths. Otherwise, typical assumptions about the quality and semantics of in-between points may not be justified. Based on our analysis we propose to modify the prior code distribution to put significantly more probability mass closer to the origin. As a result, linear interpolation paths are not only shortest paths, but they are also guaranteed to pass through high-density regions, irrespective of the dimensionality of the latent space. Experiments on standard benchmark image datasets demonstrate clear visual improvements in the quality of the generated samples and exhibit more meaningful interpolation paths.

Motivation & Objective

• To address the issue that linear interpolation in high-dimensional latent spaces of standard implicit models often passes through low-probability regions due to the concentration of a standard normal prior in a thin shell.
• To analyze the distributional mismatch between the prior and the effective distribution of interpolated points using KL divergence.
• To propose a modified prior (gamma distribution) that ensures linear interpolation paths remain in high-density regions regardless of latent dimensionality.
• To empirically validate that the new prior yields visually superior and semantically meaningful interpolation paths in image generation tasks.

Proposed method

• Replace the standard isotropic normal prior $\mathcal{N}(\mathbf{0}, \mathbf{I})$ with a gamma prior $\text{Gamma}(k, \theta)$ on the latent code magnitude, effectively concentrating mass near the origin.
• Use the gamma distribution to parameterize the radial component of the latent code, while preserving directional uniformity, resulting in a more isotropic and dense latent space near the origin.
• Formalize the KL divergence between the prior and the distribution induced on interpolated points to show dimension-independent control over distributional mismatch.
• Train GANs using the modified gamma prior and perform straight-line interpolation in the latent space, ensuring all interpolated points lie in high-probability regions.
• Evaluate interpolation quality via visual inspection and quantitative metrics such as the Latent Algebra Score (LAS) on attribute-based analogies.

Experimental results

Research questions

• RQ1 Does linear interpolation in standard implicit models with a normal prior lead to low-probability regions in high-dimensional latent spaces?
• RQ2 Can the distributional mismatch between the prior and interpolated points be formally characterized using KL divergence?
• RQ3 Does modifying the prior to concentrate mass near the origin eliminate the need for non-linear interpolation schemes like spherical interpolation?
• RQ4 Do models trained with the new prior produce visually superior and semantically meaningful interpolation paths in image generation?

Key findings

• The gamma prior significantly improves visual quality of interpolated samples in GANs, especially in high-dimensional latent spaces, compared to the standard normal prior.
• On the CelebA dataset, generators trained with the gamma prior maintain high-quality interpolation even at $d=200$, while those with the normal prior degrade rapidly with increasing dimensionality.
• The Latent Algebra Score (LAS) for attribute analogies was reduced from 0.007496 (normal prior) to 0.005638 (gamma prior), indicating better semantic structure in the latent space.
• Theoretical analysis shows that the KL divergence between the prior and interpolated point distribution is controlled independently of latent dimensionality when using the gamma prior.
• Linear interpolation with the gamma prior remains stable and effective, avoiding the instability and long paths associated with spherical interpolation.