[Paper Review] f-VAEs: Improve VAEs with Conditional Flows

Jianlin Su, Guang Wu|arXiv (Cornell University)|Sep 16, 2018

Advanced Vision and Imaging16 citations

TL;DR

This paper proposes f-VAEs, a novel generative model that integrates normalizing flows into Variational Autoencoders to improve image generation quality. By replacing the Gaussian posterior with a conditional flow-based posterior, f-VAEs achieve sharper, less blurred samples than standard VAEs while being significantly more efficient than flow-based models like Glow, converging faster and with reduced model size on the same performance level.

ABSTRACT

In this paper, we integrate VAEs and flow-based generative models successfully and get f-VAEs. Compared with VAEs, f-VAEs generate more vivid images, solved the blurred-image problem of VAEs. Compared with flow-based models such as Glow, f-VAE is more lightweight and converges faster, achieving the same performance under smaller-size architecture.

Motivation & Objective

• To address the blurred image generation problem inherent in standard VAEs, which stems from the use of a simple Gaussian posterior.
• To reduce the computational and architectural burden of flow-based models like Glow, which require deep stacks of coupling layers and extensive training.
• To unify VAEs and flow-based models into a single, generalizable framework that retains the benefits of both.
• To enable efficient, high-quality image generation using a lightweight, trainable posterior distribution based on conditional normalizing flows.

Proposed method

• The model replaces the standard VAE posterior $p(z|x)$ with a conditional flow-based posterior $p(z|x) = \int \delta(z - F_x(u)) q(u) du$, where $F_x(u)$ is a flow with parameters conditioned on the input $x$.
• The conditional flow $F_x(u)$ is implemented using invertible coupling layers, allowing exact likelihood computation and efficient posterior approximation.
• The loss function is derived as an upper bound on the KL divergence $KL(\tilde{p}(x)p(z|x) \| q(z)q(x|z))$, incorporating the log-Jacobian determinant of the flow for exact likelihood optimization.
• The encoder $E(x)$ is a deep convolutional network with squeeze operations, and the flow $F$ is a multi-scale, depth-reduced architecture inspired by Glow but with smaller kernel sizes.
• The decoder $G(z)$ is the inverse of the encoder, with a final $\tanh$ activation to produce realistic image samples.
• The framework generalizes both standard VAEs (when $F_x(u)$ reduces to a linear transformation) and unconditional flows (when $F_x(u)$ is independent of $x$), unifying both models under one objective.

Experimental results

Research questions

• RQ1 Can normalizing flows be effectively integrated into VAEs to improve image quality while maintaining computational efficiency?
• RQ2 Does a conditional flow-based posterior significantly reduce the blurriness observed in standard VAEs?
• RQ3 Can a flow-based posterior be constructed with fewer layers and smaller model size than Glow while achieving comparable or better generation quality?
• RQ4 Is there a general, differentiable framework that unifies VAEs and flow-based models under a single training objective?

Key findings

• f-VAEs successfully eliminate the blurring problem in VAE-generated images, producing sharper and more realistic samples compared to standard VAEs.
• On 64x64 CelebA-HQ, f-VAEs outperform Glow in both sample quality and training speed, achieving better results with only 7 hours of training on a single GTX 1060.
• The model achieves state-of-the-art performance on 128x128 CelebA-HQ with a smaller architecture than Glow, demonstrating superior parameter efficiency.
• Linear interpolation in the latent space of f-VAEs shows smooth, meaningful transitions between real images, indicating a well-structured and disentangled latent representation.
• The framework includes standard VAEs and unconditional flow models as special cases, confirming its theoretical generality and unifying capability.
• The ablation shows that the use of 3x3 convolutions in the encoder may limit perceptual focus, suggesting future work should explore more structured architectures like Network-in-Network or progressive growing.