[Paper Review] Ordering-Based Search: A Simple and Effective Algorithm for Learning Bayesian Networks

Marc Teyssier, Daphne Koller|arXiv (Cornell University)|Jul 4, 2012

Bayesian Modeling and Causal Inference20 references180 citations

TL;DR

This paper proposes Ordering-Based Search (OBS), a simple yet effective algorithm for learning Bayesian network structures from data. Instead of searching over the vast space of network structures, OBS searches over node orderings, efficiently finding the optimal network for each ordering—resulting in faster convergence and superior performance compared to standard greedy hill-climbing with tabu lists, while being significantly easier to implement than complex alternatives.

ABSTRACT

One of the basic tasks for Bayesian networks (BNs) is that of learning a network structure from data. The BN-learning problem is NP-hard, so the standard solution is heuristic search. Many approaches have been proposed for this task, but only a very small number outperform the baseline of greedy hill-climbing with tabu lists; moreover, many of the proposed algorithms are quite complex and hard to implement. In this paper, we propose a very simple and easy-to-implement method for addressing this task. Our approach is based on the well-known fact that the best network (of bounded in-degree) consistent with a given node ordering can be found very efficiently. We therefore propose a search not over the space of structures, but over the space of orderings, selecting for each ordering the best network consistent with it. This search space is much smaller, makes more global search steps, has a lower branching factor, and avoids costly acyclicity checks. We present results for this algorithm on both synthetic and real data sets, evaluating both the score of the network found and in the running time. We show that ordering-based search outperforms the standard baseline, and is competitive with recent algorithms that are much harder to implement.

Motivation & Objective

• To address the NP-hard problem of Bayesian network structure learning with a method that is both effective and easy to implement.
• To reduce computational complexity by shifting the search space from network structures to node orderings.
• To improve search efficiency by leveraging the fact that optimal networks for a given ordering can be computed quickly and are guaranteed acyclic.
• To achieve better score and runtime performance than baseline greedy hill-climbing with tabu lists.
• To provide a competitive alternative to complex, hard-to-implement algorithms without sacrificing performance.

Proposed method

• The algorithm performs a search over the space of possible node orderings rather than over all possible Bayesian network structures.
• For each candidate ordering, it computes the optimal Bayesian network structure consistent with that ordering using a dynamic programming approach.
• The search uses a greedy hill-climbing strategy to explore orderings, selecting the best-performing one at each step.
• The method avoids costly acyclicity checks because all networks derived from a given ordering are inherently acyclic.
• It employs a scoring function (e.g., BIC or BDeu) to evaluate the quality of the network found for each ordering.
• The algorithm uses a tabu list to prevent revisiting recently explored orderings, improving search diversity.

Experimental results

Research questions

• RQ1 Can a search over node orderings outperform traditional structure-space search in learning Bayesian networks?
• RQ2 Does restricting the search to orderings lead to better score and faster convergence than standard greedy hill-climbing with tabu lists?
• RQ3 Can a simple, efficient algorithm based on orderings achieve performance competitive with complex, state-of-the-art methods?
• RQ4 How does the branching factor and search space size compare between structure-space and ordering-space search?
• RQ5 To what extent does the ordering-based approach avoid the need for expensive acyclicity checks?

Key findings

• Ordering-Based Search outperforms the standard greedy hill-climbing with tabu lists in terms of both network score and running time on both synthetic and real-world datasets.
• The method achieves competitive scores compared to more complex algorithms, despite its simplicity and low implementation overhead.
• The search space of orderings is significantly smaller than the space of all possible Bayesian network structures, enabling more global search steps.
• The branching factor in the ordering space is lower than in the structure space, leading to more efficient exploration.
• The algorithm avoids costly acyclicity checks because all networks derived from a given ordering are guaranteed to be acyclic.
• Empirical results show that OBS achieves higher BIC scores and faster convergence than the baseline on benchmark datasets.