[Paper Review] Streaming Variational Bayes

Tamara Broderick, Nicholas Boyd+3 more|arXiv (Cornell University)|Jul 25, 2013

Bayesian Methods and Mixture Models20 references120 citations

TL;DR

This paper introduces SDA-Bayes, a streaming, distributed, asynchronous framework for approximate Bayesian inference using variational Bayes as the approximation primitive. It enables real-time posterior updates after each data batch, overcoming stochastic variational inference's limitation of requiring a pre-specified total data size, and demonstrates superior performance on LDA models for large-scale document collections.

ABSTRACT

We present SDA-Bayes, a framework for (S)treaming, (D)istributed, (A)synchronous computation of a Bayesian posterior. The framework makes streaming updates to the estimated posterior according to a user-specified approximation batch primitive. We demonstrate the usefulness of our framework, with variational Bayes (VB) as the primitive, by fitting the latent Dirichlet allocation model to two large-scale document collections. We demonstrate the advantages of our algorithm over stochastic variational inference (SVI) by comparing the two after a single pass through a known amount of data---a case where SVI may be applied---and in the streaming setting, where SVI does not apply.

Motivation & Objective

• Address the limitation of stochastic variational inference (SVI), which requires a fixed, pre-specified total data size D for posterior approximation.
• Enable true streaming Bayesian inference where the posterior is updated incrementally after each data batch, reflecting the number of documents processed so far.
• Develop a scalable framework compatible with distributed and asynchronous computation for large-scale data workloads.
• Demonstrate the framework's effectiveness on latent Dirichlet allocation (LDA) models using real-world document collections like Wikipedia and Nature.
• Overcome the sensitivity of SVI to hyperparameters like learning rate and data size estimates, which degrade performance when D is misestimated.

Proposed method

• Formulate a recursive Bayesian updating process where each new data minibatch updates the posterior using the previous posterior as a prior.
• Use variational Bayes (VB) as the user-specified approximation primitive to compute the posterior approximation after each minibatch.
• Define the streaming update rule: $ q_b(\Theta) = \mathcal{A}(C_b, q_{b-1}(Θ)) $, where $ \mathcal{A} $ applies VB to the current minibatch and the prior from the previous step.
• Ensure the framework supports distributed and asynchronous computation by decoupling the update process from sequential dependencies.
• Apply the framework to LDA by using VB to approximate the posterior over topic distributions after each minibatch of documents.
• Use a constant number of operations per data point, enabling scalability and real-time inference without revisiting past data.

Experimental results

Research questions

• RQ1 Can a Bayesian inference framework be designed to support true streaming updates, where the posterior reflects the number of documents processed so far?
• RQ2 How does the performance of SDA-Bayes compare to stochastic variational inference (SVI) when the total data size D is unknown or misestimated?
• RQ3 To what extent is SDA-Bayes robust to hyperparameter choices such as learning rate and minibatch size in a streaming context?
• RQ4 Can the framework be efficiently parallelized and scaled across distributed systems while maintaining accuracy and convergence?
• RQ5 Does using variational Bayes as the approximation primitive in a streaming setting outperform alternative methods like expectation propagation (EP) in LDA models?

Key findings

• SDA-Bayes successfully computes approximate posteriors after each minibatch, enabling real-time inference without requiring a pre-specified total data size D.
• In contrast to SVI, SDA-Bayes maintains stable performance even when the true data size D is unknown or misestimated, as shown in sensitivity analyses with D values ranging from 0.01 to 100 times the true size.
• SVI performance degrades significantly when D is misestimated, with log predictive probabilities dropping by over 1.5 units in some cases, while SDA-Bayes remains robust.
• The framework is insensitive to learning rate tuning, unlike SVI, which requires cross-validation and is highly sensitive to stepsize parameters.
• EP-based SDA-Bayes failed to converge to competitive results on LDA, taking over 91 hours on Wikipedia and 97 hours on Nature with log predictive probabilities around -7.95 and -8.02, respectively.
• SDA-Bayes with VB achieves stable and accurate inference on large-scale document collections, demonstrating scalability and adaptability in streaming environments.