[Paper Review] Perceptual Generative Autoencoders

Zijun Zhang, Ruixiang Zhang+3 more|arXiv (Cornell University)|Jun 25, 2019

Generative Adversarial Networks and Image Synthesis28 references14 citations

TL;DR

This paper proposes Perceptual Generative Autoencoders (PGA), a framework that trains generative autoencoders by matching target and generated data distributions in a learned latent space using a standard autoencoder's encoder. By enforcing consistency in both data and latent spaces with theoretically justified reconstruction losses, PGA enables non-adversarial training with maximum likelihood or VAE objectives, achieving state-of-the-art FID scores on CIFAR-10 and CelebA while improving sample quality over baseline VAEs and generalizing reversible flow models to arbitrary architectures and latent dimensions.

ABSTRACT

Modern generative models are usually designed to match target distributions directly in the data space, where the intrinsic dimension of data can be much lower than the ambient dimension. We argue that this discrepancy may contribute to the difficulties in training generative models. We therefore propose to map both the generated and target distributions to a latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space. Specifically, we enforce the consistency in both the data space and the latent space with theoretically justified data and latent reconstruction losses. The resulting generative model, which we call a perceptual generative autoencoder (PGA), is then trained with a maximum likelihood or variational autoencoder (VAE) objective. With maximum likelihood, PGAs generalize the idea of reversible generative models to unrestricted neural network architectures and arbitrary number of latent dimensions. When combined with VAEs, PGAs substantially improve over the baseline VAEs in terms of sample quality. Compared to other autoencoder-based generative models using simple priors, PGAs achieve state-of-the-art FID scores on CIFAR-10 and CelebA.

Motivation & Objective

• To address the instability and performance limitations of non-adversarial generative models like VAEs, which often produce blurry samples.
• To overcome architectural and dimensional constraints of reversible generative models that require equal data and latent space dimensions and restricted network designs.
• To improve the performance of autoencoder-based generative models by training in a latent space where data intrinsic dimensionality is lower than ambient dimensionality.
• To provide a theoretically grounded framework that ensures matching in data space by enforcing distribution matching in the latent space.
• To unify maximum likelihood and VAE objectives under a single training framework without adversarial components.

Proposed method

• The method uses a standard autoencoder to map both real and generated data into a shared latent space, where the target and generated distributions are matched.
• It enforces consistency via a data reconstruction loss and a latent reconstruction loss, with theoretical justification that minimizing latent error implies data space matching under mild assumptions.
• For maximum likelihood training (LPGA), the framework generalizes reversible flows to unrestricted architectures and arbitrary latent dimensions by removing the need for invertible layers.
• For VAE-based training (VPGA), it combines the VAE objective with the latent reconstruction loss to improve sample sharpness without auxiliary models or complex architectures.
• The framework is compatible with standard components like batch normalization, though stability issues may arise when batch norms are applied to different input distributions.
• The training objective combines data reconstruction, latent reconstruction, and either maximum likelihood or VAE objectives, with hyperparameters α, β, γ, and η controlling loss weighting.

Experimental results

Research questions

• RQ1 Can matching the target distribution in a learned latent space ensure correct distribution matching in the data space, and under what conditions?
• RQ2 Can non-adversarial generative autoencoders achieve state-of-the-art sample quality without relying on GANs or complex architectures?
• RQ3 Does training in a lower-dimensional latent space improve training stability and sample quality compared to direct data-space matching?
• RQ4 Can the framework generalize reversible flow models to arbitrary neural network architectures and latent dimensionalities?
• RQ5 How does the latent reconstruction loss contribute to performance improvements over baseline VAEs and maximum likelihood models?

Key findings

• LPGA achieves a Fréchet Inception Distance (FID) of 16.99 on MNIST, 114.19 on CIFAR-10, and 48.02 on CelebA, demonstrating strong generative performance.
• VPGA achieves state-of-the-art FID scores on CIFAR-10 and CelebA, significantly improving over baseline VAEs in sample quality without auxiliary models.
• The ablation study confirms that removing the latent reconstruction loss leads to a significant degradation in both visual quality and FID, proving its critical role in performance.
• Training with PGA is stable due to non-adversarial losses, with total loss stabilizing quickly and FID decreasing monotonically during training.
• Batch normalization can accelerate convergence and improve performance when stable, though it may introduce instability due to differing input distributions to the decoder.
• The framework unifies maximum likelihood and VAE objectives under a single training paradigm, enabling flexible and high-performing generative modeling.