[Paper Review] Pioneer Networks: Progressively Growing Generative Autoencoder

Ari Heljakka, Arno Solin+1 more|arXiv (Cornell University)|Jul 9, 2018

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper introduces PIONEER, a progressive generative autoencoder that achieves state-of-the-art image reconstruction and synthesis at 128×128 resolution without a separate GAN discriminator. By combining progressive growing with adversarial and reconstruction losses in a unified encoder-generator framework, the model enables high-fidelity image reconstruction, smooth latent space interpolation, and high-quality generation using only a single, stable training objective with spectral normalization.

ABSTRACT

We introduce a novel generative autoencoder network model that learns to encode and reconstruct images with high quality and resolution, and supports smooth random sampling from the latent space of the encoder. Generative adversarial networks (GANs) are known for their ability to simulate random high-quality images, but they cannot reconstruct existing images. Previous works have attempted to extend GANs to support such inference but, so far, have not delivered satisfactory high-quality results. Instead, we propose the Progressively Growing Generative Autoencoder (PIONEER) network which achieves high-quality reconstruction with $128{ imes}128$ images without requiring a GAN discriminator. We merge recent techniques for progressively building up the parts of the network with the recently introduced adversarial encoder-generator network. The ability to reconstruct input images is crucial in many real-world applications, and allows for precise intelligent manipulation of existing images. We show promising results in image synthesis and inference, with state-of-the-art results in CelebA inference tasks.

Motivation & Objective

• Address the lack of inference capability in GANs, which cannot reconstruct existing images despite their high-quality generation.
• Overcome the limitations of variational autoencoders (VAEs) and GAN-VAE hybrids, which fail to match the perceptual quality of state-of-the-art GANs like ProgGAN.
• Develop a unified generative autoencoder that supports both high-fidelity image reconstruction and high-quality random sampling from a smooth latent space.
• Enable progressive training of both encoder and generator networks to stabilize learning and improve performance at higher resolutions.
• Demonstrate that a separate discriminator is not necessary for achieving high-quality image generation and reconstruction when using adversarial training in a unified autoencoder framework.

Proposed method

• Propose a progressive growing training schedule for both the encoder and generator networks, starting from low-resolution features and gradually adding layers.
• Train the encoder and generator jointly using a combined loss: reconstruction loss in image space (L2) and adversarial loss in latent space using the generator’s output as real data.
• Apply spectral normalization to stabilize training and avoid mode collapse, replacing the need for a separate discriminator network.
• Use a single latent space for both inference (encoding real images) and generation (sampling from latent space), enabling smooth interpolation.
• Train the model in a fully unsupervised manner on datasets like CelebA and CelebA-HQ, without using class labels.
• Optimize the training process with a fixed architecture and minimal hyperparameter tuning, adjusting only the number of epochs per resolution stage.

Experimental results

Research questions

• RQ1 Can a generative autoencoder achieve high-quality image reconstruction and synthesis at 128×128 resolution without a separate GAN discriminator?
• RQ2 Does progressive growing of both encoder and generator improve training stability and reconstruction quality compared to standard autoencoders?
• RQ3 Can the model support smooth interpolation in the latent space for unseen image tuples, indicating a disentangled and continuous representation?
• RQ4 How does the performance of the proposed model compare to state-of-the-art GANs and autoencoder models in terms of FID and reconstruction quality at high resolutions?
• RQ5 Is spectral normalization sufficient to stabilize training and replace the need for a discriminator in a unified autoencoder setting?

Key findings

• The PIONEER model achieves state-of-the-art image reconstruction quality on CelebA at 128×128 resolution, outperforming previous autoencoder-based methods.
• The model produces high-fidelity image reconstructions with sharp details, as demonstrated by qualitative comparisons showing close alignment with input images and nearest training set matches.
• At 128×128 resolution, the model achieves a Fréchet Inception Distance (FID) of 37.50 on the LSUN bedroom dataset, indicating strong generative quality despite being trained in a fully unsupervised manner.
• The model enables smooth interpolation between latent vectors of unseen test images, demonstrating a continuous and disentangled latent space without requiring explicit supervision.
• The model generates high-quality, diverse samples at 256×256 resolution, indicating scalability beyond 128×128, though training at these resolutions was not fully completed in this work.
• The model achieves competitive performance with simpler architecture than state-of-the-art purely generative models, using only a single, unified loss with spectral normalization instead of a separate discriminator.