[Paper Review] Stochastic Structured Variational Inference

Matthew D. Hoffman, David M. Blei|arXiv (Cornell University)|Apr 16, 2014

Gaussian Processes and Bayesian InferenceComputer Science41 citations

TL;DR

This paper proposes stochastic structured variational inference, a method that relaxes the mean-field assumption by modeling arbitrary dependencies between global parameters and local latent variables. By using stochastic optimization with structured variational families, it achieves more accurate posterior approximations with reduced bias, lower sensitivity to local optima and hyperparameters, and improved parameter estimation on large datasets.

ABSTRACT

Stochastic variational inference makes it possible to approximate posterior distributions induced by large datasets quickly using stochastic optimization. The algorithm relies on the use of fully factorized variational distributions. However, this independence approximation limits the fidelity of the posterior approximation, and introduces local optima. We show how to relax the mean-field approximation to allow arbitrary dependencies between global parameters and local hidden variables, producing better parameter estimates by reducing bias, sensitivity to local optima, and sensitivity to hyperparameters.

Motivation & Objective

• To address the limitations of mean-field variational inference in approximating complex posterior distributions.
• To reduce bias and sensitivity to local optima and hyperparameters in large-scale Bayesian inference.
• To enable flexible, structured variational approximations that capture dependencies between global parameters and local latent variables.
• To develop a scalable stochastic optimization framework compatible with large datasets.

Proposed method

• Proposes a structured variational family that explicitly models dependencies between global parameters and local latent variables, moving beyond fully factorized approximations.
• Uses stochastic optimization to scale the inference procedure to large datasets by sampling mini-batches of data.
• Employs a reparameterization trick to enable gradient-based optimization of the variational parameters in the structured family.
• Derives a lower bound on the marginal likelihood (evidence lower bound) that accounts for the structured dependencies in the variational distribution.
• Introduces a stochastic approximation of the variational lower bound that enables online learning and efficient optimization.

Experimental results

Research questions

• RQ1 Can structured variational inference with arbitrary dependencies between global and local variables improve posterior approximation quality?
• RQ2 Does relaxing the mean-field assumption reduce bias and sensitivity to hyperparameters in large-scale Bayesian models?
• RQ3 Can stochastic optimization be effectively applied to structured variational families without sacrificing scalability?
• RQ4 How do the proposed methods compare to standard mean-field variational inference in terms of convergence and parameter estimation accuracy?

Key findings

• The structured variational approach achieves significantly lower approximation bias compared to mean-field variational inference.
• The method demonstrates reduced sensitivity to local optima and hyperparameter choices in experimental evaluations.
• Stochastic optimization enables scalable inference on large datasets while maintaining high-quality posterior approximations.
• The proposed method improves parameter estimation accuracy by capturing meaningful dependencies between global and local variables.