[Paper Review] Variational Variance: Simple and Reliable Predictive Variance Parameterization

Andrew Stirn, David A. Knowles|arXiv (Cornell University)|Jun 8, 2020

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a variational approach to modeling predictive variance in variational autoencoders (VAEs) by treating variance as a latent variable with inferred mixture proportions, enabling robust, heteroscedastic uncertainty estimation. The method improves generation quality and generalization in both unsupervised and supervised continuous data modeling by synergizing with existing VAE theory and Empirical Bayes techniques.

ABSTRACT

An often overlooked sleight of hand performed with variational autoencoders (VAEs), which has proliferated the literature, is to misrepresent the posterior predictive (decoder) distribution's expectation as a sample from that distribution. Jointly modeling the mean and variance for a normal predictive distribution can result in fragile optimization where the ultimately learned parameters can be ineffective at generating realistic samples. The two most common principled methods to avoid this problem are to either fix the variance or use the single-parameter Bernoulli distribution--both have drawbacks, however. Unfortunately, the problem of jointly optimizing mean and variance networks affects not only unsupervised modeling of continuous data (a taxonomy for many VAE applications) but also regression tasks. To date, only a handful of papers have attempted to resolve these difficulties. In this article, we propose an alternative and attractively simple solution: treat predictive variance variationally. Our approach synergizes with existing VAE-specific theoretical results and, being probabilistically principled, provides access to Empirical Bayes and other such techniques that utilize the observed data to construct well-informed priors. We extend the VAMP prior, which assumes a uniform mixture, by inferring mixture proportions and assignments. This extension amplifies our ability to accurately capture heteroscedastic variance. Notably, our methods experimentally outperform existing techniques on supervised and unsupervised modeling of continuous data.

Motivation & Objective

• To address the common but problematic practice in VAEs of misrepresenting the posterior predictive mean as a sample, which undermines reliable uncertainty estimation.
• To overcome the fragility of joint optimization of mean and variance in normal likelihoods, especially in continuous and regression tasks.
• To develop a probabilistically principled alternative to fixed-variance or Bernoulli-based variance modeling, which suffer from significant limitations.
• To extend the VAMP prior by inferring mixture proportions and assignments, enabling accurate modeling of heteroscedastic variance.

Proposed method

• Treat the predictive variance as a latent variable modeled through a variational distribution, allowing end-to-end optimization of uncertainty.
• Extend the VAMP prior by inferring mixture proportions and component assignments, enabling flexible, data-driven modeling of heteroscedastic variance.
• Use a mixture-of-experts style formulation where each component corresponds to a different variance level, with mixture weights learned from data.
• Integrate the variance modeling into the VAE objective via a variational lower bound that accounts for uncertainty in both mean and variance.
• Leverage Empirical Bayes techniques by using observed data to inform prior distributions over variance, improving prior calibration.
• Apply the method to both unsupervised VAEs and supervised regression tasks, demonstrating broad applicability.

Experimental results

Research questions

• RQ1 Can a variational parameterization of predictive variance improve the reliability and realism of generated samples in VAEs?
• RQ2 How does joint optimization of mean and variance networks affect model stability and performance in continuous data modeling?
• RQ3 To what extent can inferred mixture proportions and assignments in a VAMP-like prior improve heteroscedastic uncertainty estimation?
• RQ4 Does the proposed method outperform fixed-variance and Bernoulli-based variance modeling in both unsupervised and supervised settings?
• RQ5 Can the method be effectively combined with Empirical Bayes techniques to improve prior calibration and generalization?

Key findings

• The proposed method achieves superior generation quality compared to existing approaches on unsupervised modeling of continuous data, particularly in capturing complex variance structures.
• The model demonstrates improved robustness in regression tasks by learning heteroscedastic uncertainty, outperforming baselines with fixed or Bernoulli variance.
• Inferring mixture proportions and assignments in the VAMP prior significantly enhances the model's ability to capture data-driven variance patterns.
• The method enables effective use of Empirical Bayes techniques by providing well-informed priors derived from observed data.
• Experiments confirm that the approach is both stable and scalable, with consistent improvements across diverse continuous data modeling tasks.