[Paper Review] Latent Variable Modelling Using Variational Autoencoders: A survey

Vasanth Kalingeri|arXiv (Cornell University)|Jun 20, 2022

Computational and Text Analysis Methods4 citations

TL;DR

This survey provides a comprehensive overview of variational autoencoders (VAEs) as a framework for latent variable modeling, explaining their theoretical foundations, optimization techniques, and recent advancements. It details how VAEs use variational inference to approximate intractable posterior distributions, compares reparameterization and score function gradients, and reviews methods to improve model capacity, reduce gradient variance, and model discrete latent variables.

ABSTRACT

A probability distribution allows practitioners to uncover hidden structure in the data and build models to solve supervised learning problems using limited data. The focus of this report is on Variational autoencoders, a method to learn the probability distribution of large complex datasets. The report provides a theoretical understanding of variational autoencoders and consolidates the current research in the field. The report is divided into multiple chapters, the first chapter introduces the problem, describes variational autoencoders and identifies key research directions in the field. Chapters 2, 3, 4 and 5 dive into the details of each of the key research areas. Chapter 6 concludes the report and suggests directions for future work. A reader who has a basic idea of machine learning but wants to learn about general themes in machine learning research can benefit from the report. The report explains central ideas on learning probability distributions, what people did to make this tractable and goes into details around how deep learning is currently applied. The report also serves a gentle introduction for someone looking to contribute to this sub-field.

Motivation & Objective

• To provide a unified theoretical and practical understanding of variational autoencoders (VAEs) as a scalable method for learning complex data distributions.
• To identify and analyze key research directions in VAEs, including improved inference, better decoders, and modeling of discrete latent states.
• To explain the trade-offs between reparameterization and score function gradients in VAE training, focusing on variance reduction and optimization stability.
• To survey recent techniques such as normalizing flows, Bayesian non-parametrics, and control variates that enhance VAE performance and expressivity.
• To serve as a foundational resource for researchers entering the field of deep generative modeling with VAEs.

Proposed method

• Uses the evidence lower bound (ELBO) as the objective function to approximate the marginal log-likelihood of observed data under a latent variable model.
• Applies the reparameterization trick to enable low-variance backpropagation through stochastic nodes by expressing latent variables as deterministic functions of noise variables.
• Employs score function gradients (REINFORCE) for non-reparameterizable distributions, particularly for discrete latent variables, and uses control variates to reduce gradient variance.
• Introduces hierarchical inference and normalizing flows to improve the flexibility and expressivity of the approximate posterior distribution.
• Utilizes variational Gaussian processes and Bayesian non-parametric methods to allow non-parametric modeling of the latent space and improve generalization.
• Applies importance sampling and Monte Carlo objectives to refine the ELBO approximation and improve training stability and convergence.

Experimental results

Research questions

• RQ1 How can variational autoencoders be improved to better model complex, high-dimensional data distributions such as images and text?
• RQ2 What are the trade-offs between reparameterization gradients and score function gradients in VAE optimization, and how can gradient variance be effectively reduced?
• RQ3 How can discrete latent variables be modeled in VAEs when the reparameterization trick is not applicable?
• RQ4 What techniques enhance the expressivity of the inference network and decoder in VAEs without increasing computational cost?
• RQ5 How do normalizing flows and Bayesian non-parametric methods improve the quality and flexibility of the learned latent representations?

Key findings

• The reparameterization trick enables low-variance, differentiable gradients through the latent variable sampling process, making VAE training stable and efficient for continuous latent variables.
• Score function gradients, while applicable to non-reparameterizable distributions like discrete variables, suffer from high variance, which can be mitigated using control variates and Rao–Blackwellization.
• Normalizing flows significantly improve the flexibility of the approximate posterior by transforming a simple base distribution into a complex one through invertible, differentiable mappings.
• Hierarchical inference and variational Gaussian processes allow for more expressive and adaptive posterior approximations, improving model capacity and generalization.
• Importance sampling and Monte Carlo objectives provide tighter bounds on the marginal likelihood, leading to better optimization and improved generation quality.
• Continuous approximations of discrete latent variables, such as the Gumbel-Softmax trick, enable end-to-end training of VAEs with discrete latents by providing differentiable sampling.