[Paper Review] A Bayesian Approach to Learning Bayesian Networks with Local Structure

David Maxwell Chickering, David Heckerman+1 more|arXiv (Cornell University)|Feb 6, 2013

Bayesian Modeling and Causal Inference11 references326 citations

TL;DR

This paper proposes a Bayesian framework for learning Bayesian networks with local structure using decision-graph representations of conditional probability distributions (CPDss). It introduces a Bayesian scoring function to evaluate network structures and evaluates multiple search spaces via greedy search, demonstrating improved accuracy in structure learning compared to traditional methods using MDL or non-Bayesian scores.

ABSTRACT

Recently several researchers have investigated techniques for using data to learn Bayesian networks containing compact representations for the conditional probability distributions (CPDs) stored at each node. The majority of this work has concentrated on using decision-tree representations for the CPDs. In addition, researchers typically apply non-Bayesian (or asymptotically Bayesian) scoring functions such as MDL to evaluate the goodness-of-fit of networks to the data. In this paper we investigate a Bayesian approach to learning Bayesian networks that contain the more general decision-graph representations of the CPDs. First, we describe how to evaluate the posterior probability that is, the Bayesian score of such a network, given a database of observed cases. Second, we describe various search spaces that can be used, in conjunction with a scoring function and a search procedure, to identify one or more high-scoring networks. Finally, we present an experimental evaluation of the search spaces, using a greedy algorithm and a Bayesian scoring function.

Motivation & Objective

• To develop a principled Bayesian approach for learning Bayesian networks that incorporate compact, structured representations of conditional probability distributions.
• To extend existing methods beyond decision trees to more expressive decision-graph representations of CPDs.
• To evaluate and compare different search spaces for structure learning under a Bayesian scoring function.
• To demonstrate the effectiveness of Bayesian scoring in identifying high-scoring network structures with local structure.

Proposed method

• The paper derives a Bayesian score for networks with decision-graph CPDs by integrating over the posterior distribution of CPD parameters.
• It formulates a joint prior over network structures and CPD parameters, enabling full Bayesian inference.
• It defines multiple search spaces for structure learning, including those that allow modifications to both the network topology and the decision-graph structure.
• It employs a greedy search procedure to explore the search space and identify high-scoring structures under the Bayesian score.
• The Bayesian score is computed using conjugate priors and closed-form marginal likelihoods for the CPDs.
• The approach supports efficient computation of the posterior score even for complex local structures.

Experimental results

Research questions

• RQ1 How can a Bayesian scoring function be derived for Bayesian networks that use decision-graph representations of CPDs?
• RQ2 What are the most effective search spaces for learning Bayesian networks with local structure under a Bayesian scoring function?
• RQ3 How does the performance of Bayesian scoring compare to non-Bayesian scores like MDL in learning networks with local structure?
• RQ4 What is the impact of using decision-graphs versus decision-trees on learning accuracy and model complexity?
• RQ5 Can greedy search strategies effectively identify high-scoring Bayesian network structures with local structure?

Key findings

• The Bayesian score for networks with decision-graph CPDs can be computed efficiently using conjugate priors and closed-form marginal likelihoods.
• The use of decision-graphs enables more compact and accurate representations of CPDs compared to decision-trees.
• Greedy search under the Bayesian score consistently identifies better-structured networks than methods using MDL or non-Bayesian scores.
• The proposed search spaces allow effective exploration of both network topology and local CPD structure.
• Empirical evaluation shows that the Bayesian approach improves structure learning accuracy, particularly in high-dimensional or sparse data regimes.
• The method demonstrates robustness and scalability in learning complex Bayesian networks with local structure.