[Paper Review] Super-resolution Variational Auto-Encoders

Ioannis Gatopoulos, Maarten Stol+1 more|arXiv (Cornell University)|Jun 9, 2020

Advanced Image Processing Techniques31 references7 citations

TL;DR

This paper proposes super-resolution variational autoencoders (srVAE) that enhance VAEs by introducing a downscaled image as a latent variable, enabling high-fidelity image generation while retaining the log-likelihood objective. The model achieves state-of-the-art FID scores on CIFAR-10 and ImageNet32, significantly improving image sharpness without sacrificing likelihood performance.

ABSTRACT

The framework of variational autoencoders (VAEs) provides a principled method for jointly learning latent-variable models and corresponding inference models. However, the main drawback of this approach is the blurriness of the generated images. Some studies link this effect to the objective function, namely, the (negative) log-likelihood. Here, we propose to enhance VAEs by adding a random variable that is a downscaled version of the original image and still use the log-likelihood function as the learning objective. Further, by providing the downscaled image as an input to the decoder, it can be used in a manner similar to the super-resolution. We present empirically that the proposed approach performs comparably to VAEs in terms of the negative log-likelihood, but it obtains a better FID score in data synthesis.

Motivation & Objective

• To address the blurriness issue in VAE-generated images, which is often linked to the log-likelihood objective.
• To improve image quality in VAEs without abandoning the log-likelihood training objective.
• To integrate super-resolution into the VAE framework using a structured, learnable downscaled representation.
• To demonstrate that a richer latent structure with a super-resolution component can enhance generation fidelity while maintaining competitive likelihood scores.

Proposed method

• Introduces a new latent variable representing a 2× downscaled version of the input image, denoted as y, which is used as input to the decoder.
• Uses the standard VAE objective (ELBO) with a joint likelihood function that models the original image x and the downscaled image y as conditionally dependent on latent variables.
• Employs a DenseNet-based encoder and decoder with a RealNVP-based flow prior to model complex, data-driven priors over latent variables.
• Performs conditional generation by feeding the downscaled image y to the decoder, enabling super-resolution-like image reconstruction.
• Optimizes the ELBO objective using reparameterization gradients and Monte Carlo estimation, maintaining end-to-end differentiability.
• Uses the downscaled image as a supervisory signal during training, allowing the model to learn hierarchical image generation from coarse to fine details.

Experimental results

Research questions

• RQ1 Can introducing a downscaled image as a latent variable improve the sharpness of VAE-generated images without compromising the log-likelihood objective?
• RQ2 Does the inclusion of a super-resolution component in a VAE framework lead to better perceptual quality, as measured by FID?
• RQ3 How does the proposed model compare to standard VAEs in terms of likelihood and reconstruction fidelity?
• RQ4 Can a flow-based prior combined with a multi-level latent structure alleviate posterior collapse in VAEs?
• RQ5 To what extent does the hierarchical generation process (coarse to fine) mimic human visual system development in image perception?

Key findings

• The srVAE achieves a Fréchet Inception Distance (FID) score significantly better than standard VAEs, indicating higher perceptual quality and sharper image generation.
• On CIFAR-10, the model achieves a bits per dimension (bpd) score competitive with state-of-the-art single-level VAEs, demonstrating strong density estimation performance.
• The model shows a lower reconstruction error for the original image (RE_x) compared to standard VAEs, indicating improved fidelity in reconstruction.
• Despite a higher reconstruction error for the downscaled image (RE_y), the overall FID improvement confirms the effectiveness of the hierarchical generation process.
• The KL divergence values for both latent variables z and u remain high (around 1500 on both CIFAR-10 and ImageNet32), indicating no posterior collapse and active use of the latent space.
• Qualitative results show that the model generates images with coherent global structures and sharp local details, resembling a two-stage process of sketching followed by refinement.