[Paper Review] Expectation Propagation for approximate Bayesian inference

Thomas P. Minka|arXiv (Cornell University)|Jan 10, 2013

Bayesian Modeling and Causal Inference15 references1,457 citations

TL;DR

This paper introduces Expectation Propagation (EP), a deterministic approximate inference method in Bayesian networks that unifies and extends techniques like loopy belief propagation and assumed-density filtering. By iteratively refining approximate beliefs using sufficient statistics (e.g., mean and variance), EP achieves superior accuracy over Laplace’s method, variational Bayes, and Monte Carlo methods at comparable computational cost, particularly in hybrid and correlated models.

ABSTRACT

This paper presents a new deterministic approximation technique in Bayesian networks. This method, "Expectation Propagation", unifies two previous techniques: assumed-density filtering, an extension of the Kalman filter, and loopy belief propagation, an extension of belief propagation in Bayesian networks. All three algorithms try to recover an approximate distribution which is close in KL divergence to the true distribution. Loopy belief propagation, because it propagates exact belief states, is useful for a limited class of belief networks, such as those which are purely discrete. Expectation Propagation approximates the belief states by only retaining certain expectations, such as mean and variance, and iterates until these expectations are consistent throughout the network. This makes it applicable to hybrid networks with discrete and continuous nodes. Expectation Propagation also extends belief propagation in the opposite direction - it can propagate richer belief states that incorporate correlations between nodes. Experiments with Gaussian mixture models show Expectation Propagation to be convincingly better than methods with similar computational cost: Laplace's method, variational Bayes, and Monte Carlo. Expectation Propagation also provides an efficient algorithm for training Bayes point machine classifiers.

Motivation & Objective

• To develop a unified, deterministic approximation technique for Bayesian inference in graphical models.
• To extend belief propagation to handle continuous and hybrid networks with correlated variables.
• To improve accuracy over existing low-cost methods like Laplace’s method and variational Bayes.
• To provide an efficient algorithm for training Bayes point machine classifiers.
• To enable approximate inference in models where exact inference is intractable.

Proposed method

• EP approximates the true posterior distribution by iteratively refining a set of tractable, factorized approximations.
• It uses moment matching to align sufficient statistics (e.g., mean and variance) of the approximate factors with those of the true posterior.
• The algorithm performs message passing in a factor graph, updating approximate factors based on local evidence and global consistency.
• EP handles both discrete and continuous nodes, making it suitable for hybrid Bayesian networks.
• It propagates richer belief states that capture correlations between variables, unlike standard belief propagation.
• The method minimizes KL divergence between the approximate and true posterior distributions through iterative refinement.

Experimental results

Research questions

• RQ1 Can a deterministic approximation method unify and outperform existing techniques like loopy belief propagation and variational inference?
• RQ2 How can approximate inference be extended to hybrid Bayesian networks with both discrete and continuous variables?
• RQ3 Can EP achieve higher accuracy than Laplace’s method and variational Bayes while maintaining similar computational cost?
• RQ4 How does EP perform in modeling correlated variables compared to standard belief propagation?
• RQ5 Can EP be efficiently applied to real-world learning tasks such as training Bayes point machines?

Key findings

• EP significantly outperforms Laplace’s method, variational Bayes, and Monte Carlo methods in terms of approximation accuracy for Gaussian mixture models.
• The method achieves better predictive performance with comparable computational complexity, demonstrating its efficiency.
• EP successfully handles hybrid Bayesian networks, extending the applicability of belief propagation beyond purely discrete models.
• By retaining higher-order moments and correlations, EP provides more accurate posterior approximations than mean-field methods.
• EP enables efficient training of Bayes point machine classifiers, showing practical utility in machine learning applications.
• Empirical results confirm that EP converges to consistent sufficient statistics across the network, validating its iterative refinement process.