[Paper Review] Bayesian Conditional Density Filtering

Shaan Qamar, Rajarshi Guhaniyogi+1 more|arXiv (Cornell University)|Jan 15, 2014

Gaussian Processes and Bayesian Inference38 references15 citations

TL;DR

This paper introduces Bayesian Conditional Density Filtering (C-DF), a scalable online Bayesian inference method that uses surrogate conditional sufficient statistics (SCSS) to enable efficient MCMC sampling in streaming, high-dimensional data settings. By propagating SCSS instead of raw data, C-DF reduces memory and computational costs while maintaining accurate posterior approximation and convergence to the true posterior as data accumulate.

ABSTRACT

We propose a Conditional Density Filtering (C-DF) algorithm for efficient online Bayesian inference. C-DF adapts MCMC sampling to the online setting, sampling from approximations to conditional posterior distributions obtained by propagating surrogate conditional sufficient statistics (a function of data and parameter estimates) as new data arrive. These quantities eliminate the need to store or process the entire dataset simultaneously and offer a number of desirable features. Often, these include a reduction in memory requirements and runtime and improved mixing, along with state-of-the-art parameter inference and prediction. These improvements are demonstrated through several illustrative examples including an application to high dimensional compressed regression. Finally, we show that C-DF samples converge to the target posterior distribution asymptotically as sampling proceeds and more data arrives.

Motivation & Objective

• To address the computational infeasibility of traditional MCMC methods in big data settings where full data storage and likelihood evaluations are prohibitive.
• To enable efficient online Bayesian inference for high-dimensional streaming data by adapting MCMC to sequential data arrival.
• To reduce memory and runtime costs while preserving posterior accuracy and improving MCMC mixing in large-scale Bayesian models.
• To establish theoretical convergence of C-DF samples to the true posterior as more data arrive.
• To extend MCMC scalability to models with high-dimensional parameter spaces and complex likelihoods using variational approximation and surrogate statistics.

Proposed method

• C-DF uses surrogate conditional sufficient statistics (SCSS) to summarize data and parameter estimates, avoiding storage of the entire dataset.
• It applies fixed-point iteration to update approximate posterior parameters for the conditional distribution at each time step.
• The method partitions parameters into groups, updating them iteratively using SCSS propagated from previous time points.
• For each data shard, it updates sufficient statistics like ${\boldsymbol{C}}_{1,1}^{t}$, ${\boldsymbol{C}}_{1,2}^{t}$, and ${\boldsymbol{C}}_{1,3}^{t}$ to maintain sufficient information for posterior updates.
• It employs variational approximation to define the conditional posterior, enabling efficient computation without full likelihood evaluation at each step.
• The algorithm alternates between updating mean and covariance parameters for the main model and hyperparameters of the inverse-gamma and inverse-gaussian priors.

Experimental results

Research questions

• RQ1 Can MCMC sampling be adapted to online, streaming data settings without storing the entire dataset?
• RQ2 Can surrogate conditional sufficient statistics (SCSS) effectively replace full data processing to reduce memory and computational costs in Bayesian inference?
• RQ3 Does the C-DF algorithm maintain convergence to the true posterior as data accumulate over time?
• RQ4 Can C-DF achieve improved mixing and accuracy in high-dimensional regression models compared to standard MCMC or subsampling methods?
• RQ5 How well does C-DF perform in compressed regression and other high-dimensional settings with streaming data?

Key findings

• C-DF significantly reduces memory and runtime requirements by propagating SCSS instead of raw data or full sufficient statistics.
• The algorithm achieves improved MCMC mixing compared to standard MCMC, especially in high-dimensional models.
• C-DF samples converge asymptotically to the true posterior distribution as more data arrive, ensuring theoretical validity.
• Empirical results show state-of-the-art performance in parameter inference and prediction, particularly in high-dimensional compressed regression.
• The method maintains accuracy even when data are processed in small, sequential shards, demonstrating robustness to data streaming.
• Fixed-point iteration on SCSS enables efficient parameter updates without full likelihood evaluations, making it scalable to large datasets.