[Paper Review] Bayesian Conditional Density Filtering for Big Data.

Rajarshi Guhaniyogi, Shaan Qamar+1 more|arXiv (Cornell University)|Jan 15, 2014

Bayesian Methods and Mixture Models15 references12 citations

TL;DR

This paper proposes Conditional Density Filtering (C-DF), an online Bayesian inference algorithm that uses Gibbs sampling with tracked surrogate sufficient statistics to enable efficient, scalable posterior sampling without storing all data. C-DF asymptotically converges to the exact posterior and achieves competitive performance with batch methods in streaming compressed factor regression.

ABSTRACT

We propose a Conditional Density Filtering (C-DF) algorithm for efficient online Bayesian inference. C-DF adapts Gibbs sampling to the online setting, sampling from approximations to conditional posterior distributions obtained by tracking of surrogate conditional sufficient statistics as new data arrive. This tracking eliminates the need to store or process the entire data set simultaneously. We show that C-DF samples converge to the exact posterior distribution asymptotically, as sampling proceeds and more data arrive over time. We provide several motivating examples, and consider an application to compressed factor regression for streaming data, illustrating competitive performance with batch algorithms that use all of the data.

Motivation & Objective

• To enable efficient online Bayesian inference for big data by avoiding storage and reprocessing of entire datasets.
• To develop a method that maintains posterior sampling accuracy in streaming settings without batch reprocessing.
• To track surrogate conditional sufficient statistics to approximate posterior distributions incrementally as new data arrive.
• To ensure asymptotic convergence of samples to the exact posterior distribution over time.
• To demonstrate competitive performance against batch algorithms in real-world streaming applications like compressed factor regression.

Proposed method

• C-DF uses Gibbs sampling adapted to the online setting by maintaining surrogate conditional sufficient statistics as data stream in.
• It approximates conditional posterior distributions using these tracked statistics, eliminating the need to reprocess historical data.
• The algorithm incrementally updates sufficient statistics upon arrival of new data points, enabling real-time inference.
• C-DF ensures asymptotic convergence to the true posterior by refining estimates with increasing data volume.
• The method supports complex models like compressed factor regression by maintaining conditional posterior approximations efficiently.
• It enables scalable inference by avoiding full data storage and recomputation at each step.

Experimental results

Research questions

• RQ1 Can online Bayesian inference be performed efficiently without storing or reprocessing the entire dataset?
• RQ2 How can surrogate sufficient statistics be used to maintain accurate posterior approximations in streaming settings?
• RQ3 Does the C-DF algorithm converge asymptotically to the exact posterior distribution as more data arrive?
• RQ4 How does C-DF perform in comparison to batch inference methods in terms of accuracy and scalability?
• RQ5 Can C-DF be effectively applied to high-dimensional streaming regression tasks such as compressed factor regression?

Key findings

• C-DF achieves asymptotic convergence to the exact posterior distribution as the number of data points increases.
• The algorithm maintains high sampling accuracy without requiring storage or reprocessing of the full dataset.
• C-DF demonstrates competitive performance with batch algorithms in compressed factor regression on streaming data.
• The use of surrogate sufficient statistics enables scalable, real-time Bayesian inference in big data settings.
• The method effectively handles online learning in complex models by tracking only necessary sufficient statistics.
• Empirical results confirm that C-DF maintains posterior approximation quality while significantly reducing computational overhead.