[Paper Review] Hidden Talents of the Variational Autoencoder

Bin Dai, Yu Wang+3 more|arXiv (Cornell University)|Jun 16, 2017

Generative Adversarial Networks and Image Synthesis45 references17 citations

TL;DR

This paper reveals that variational autoencoders (VAEs) possess an intrinsic robustness to sparse outliers in data, functioning as a natural extension of robust PCA for nonlinear, low-dimensional manifolds. Through tractable analytical cases, it demonstrates that VAEs minimize reconstruction error under gross corruptions, effectively dismissing outliers while preserving the underlying manifold structure.

ABSTRACT

Variational autoencoders (VAE) represent a popular, flexible form of deep generative model that can be stochastically fit to samples from a given random process using an information-theoretic variational bound on the true underlying distribution. Once so-obtained, the model can be putatively used to generate new samples from this distribution, or to provide a low-dimensional latent representation of existing samples. While quite effective in numerous application domains, certain important mechanisms which govern the behavior of the VAE are obfuscated by the intractable integrals and resulting stochastic approximations involved. Moreover, as a highly non-convex model, it remains unclear exactly how minima of the underlying energy relate to original design purposes. We attempt to better quantify these issues by analyzing a series of tractable special cases of increasing complexity. In doing so, we unveil interesting connections with more traditional dimensionality reduction models, as well as an intrinsic yet underappreciated propensity for robustly dismissing sparse outliers when estimating latent manifolds. With respect to the latter, we demonstrate that the VAE can be viewed as the natural evolution of recent robust PCA models, capable of learning nonlinear manifolds of unknown dimension obscured by gross corruptions.

Motivation & Objective

• To investigate the hidden robustness properties of variational autoencoders (VAEs) beyond standard generative modeling.
• To analyze how VAEs behave under gross corruptions and sparse outliers in the data manifold.
• To establish a theoretical link between VAEs and robust PCA, especially in nonlinear, low-dimensional settings.
• To quantify the VAE's ability to learn latent manifolds obscured by sparse, high-magnitude outliers.
• To demonstrate that VAEs can serve as a natural evolution of robust PCA models for nonlinear data

Proposed method

• The authors analyze tractable special cases of VAEs with piecewise linear decoder functions and Gaussian latent variables to simplify the variational objective.
• They define a quantization-based decoder where the mean output is set to the nearest training sample based on latent code, enabling exact computation of the reconstruction term.
• A key component is the use of a piecewise linear function h(z) that maps latent codes to training samples, ensuring that only nearby samples are reconstructed.
• The method employs Chebyshev's inequality to bound the probability of incorrect quantization, which in turn bounds the expected reconstruction error.
• The analysis derives an upper bound on the VAE objective that scales with the inverse of the minimum inter-latent-distance and the maximal pairwise data distance.
• By setting hyperparameters λx and λz to α and taking α→0, the bound becomes unbounded from below, showing that the model can achieve arbitrarily low objective values under favorable conditions.

Experimental results

Research questions

• RQ1 Can VAEs robustly estimate low-dimensional manifolds when data contain sparse, high-magnitude outliers?
• RQ2 How does the VAE objective behave under gross corruptions, and does it naturally suppress outlier influence?
• RQ3 To what extent does the VAE’s optimization landscape resemble that of robust PCA models?
• RQ4 What theoretical guarantees can be derived for VAEs in the presence of sparse outliers in the training data?
• RQ5 Is the VAE’s latent space optimization inherently biased toward preserving the underlying manifold while rejecting outliers?

Key findings

• The VAE objective can be bounded below by a term that depends on the inverse of the minimum distance between latent codes, implying that well-separated codes lead to lower reconstruction error.
• The expected reconstruction error is bounded by η/λx times the probability of incorrect quantization, where η is the maximum pairwise distance between data points.
• Using Chebyshev’s inequality, the probability of incorrect quantization is bounded by λz/ρ², where ρ is half the minimum squared distance between latent means.
• As the variance parameter α = λx = λz approaches zero, the upper bound on the VAE objective becomes unbounded from below, indicating that the model can achieve arbitrarily low loss under favorable latent configurations.
• The analysis shows that VAEs inherently resist sparse outliers because the reconstruction error for corrupted samples is penalized only when the latent code maps to a wrong data point, which is unlikely under proper separation.
• The VAE is shown to be a natural generalization of robust PCA to nonlinear, unknown-dimension manifolds, with the ability to learn the true manifold even when corrupted by sparse outliers.