[Paper Review] Novel Bounds on Marginal Probabilities

Joris M. Mooij, Hilbert J. Kappen|ArXiv.org|Jan 24, 2008

Bayesian Modeling and Causal Inference15 references3 citations

TL;DR

This paper introduces two novel methods—BoxProp-SubT and BoxProp-SAWT—that compute tight, computationally efficient bounds on single-variable marginal probabilities in discrete factor graphs. By propagating bounding boxes over subtrees and self-avoiding walk trees, respectively, the methods simultaneously bound both exact marginals and Belief Propagation beliefs, offering improved accuracy and lower computation time than existing bounds, especially for low-cardinality variables and strong interactions.

ABSTRACT

We derive two related novel bounds on single-variable marginal probability distributions in factor graphs with discrete variables. The first method propagates bounds over a subtree of the factor graph rooted in the variable, and the second method propagates bounds over the self-avoiding walk tree starting at the variable. By construction, both methods not only bound the exact marginal probability distribution of a variable, but also its approximate Belief Propagation marginal (``belief''). Thus, apart from providing a practical means to calculate bounds on marginals, our contribution also lies in an increased understanding of the error made by Belief Propagation. Empirically, we show that our bounds often outperform existing bounds in terms of accuracy and/or computation time. We also show that our bounds can yield nontrivial results for medical diagnosis inference problems.

Motivation & Objective

• To develop tighter, practical bounds on single-variable marginal probabilities in discrete factor graphs.
• To quantify the error of Belief Propagation by bounding both exact marginals and BP beliefs simultaneously.
• To reduce computational cost while maintaining or improving bound tightness compared to existing methods.
• To enable reliable error estimation in approximate inference without requiring exact solutions.
• To support real-world applications such as medical diagnosis where tight, fast bounds are essential.

Proposed method

• Propagates bounding boxes (smallest enclosing boxes for probability measures) over a subtree rooted at the target variable to compute marginal bounds.
• Uses the self-avoiding walk tree (SAWT) as a computation tree to propagate boxes, capturing more complex dependencies than subtree propagation.
• Applies convexity and normalization/multiplication operations to maintain valid probability bounds during propagation.
• Employs a local bound on the marginal of a variable given its neighbors, using a box representation to track bounds efficiently.
• Truncates subtrees and SAW trees to control computation time, trading off some tightness for scalability.
• Combines the bounds from multiple methods via intersection or tightest bound to improve overall accuracy.

Experimental results

Research questions

• RQ1 Can we derive tighter bounds on single-variable marginals in factor graphs that also bound the Belief Propagation belief?
• RQ2 How can we efficiently compute bounds with low computational cost, especially for low-cardinality variables?
• RQ3 Can we improve upon existing bounds in terms of tightness and computation time for real-world inference problems?
• RQ4 How do the bounds behave under strong interactions and higher-order factor graphs?
• RQ5 Can the bounds be extended to continuous variables or generalized beyond pairwise interactions?

Key findings

• BoxProp-SubT and BoxProp-SAWT outperform existing bounds in terms of tightness and computation time, especially for binary or low-cardinality variables.
• For pairwise interactions, BoxProp-SAWT yields the tightest bounds with linear scaling in network size when truncated to fixed depth.
• The bounds are simultaneously valid for both exact marginals and BP beliefs, enabling direct error quantification of BP.
• Computation time is predictable and independent of interaction strengths, unlike iterative methods such as BoundProp or TRW.
• BoxProp-SubT is superior for higher-order interactions, where Ihler-SAWT cannot be easily extended.
• The bounds are effective for real-world medical diagnosis problems, yielding nontrivial and useful results.