[Paper Review] A Parallel Algorithm for Exact Bayesian Structure Discovery in Bayesian Networks

Yetian Chen, Jin Tian+2 more|arXiv (Cornell University)|Aug 7, 2014

Bayesian Modeling and Causal Inference20 references3 citations

TL;DR

This paper presents ParaREBEL, a parallel algorithm that enables exact Bayesian structure discovery in Bayesian networks with optimal space and near-optimal time efficiency by exploiting the hypercube structure of dynamic programming subproblems. It achieves scalability on up to 2048 processors and successfully discovers the yeast pheromone response pathway in a 33-variable dataset, demonstrating the first practical solution for exact posterior probability computation of all edges in large BNs.

ABSTRACT

Exact Bayesian structure discovery in Bayesian networks requires exponential time and space. Using dynamic programming (DP), the fastest known sequential algorithm computes the exact posterior probabilities of structural features in $O(2(d+1)n2^n)$ time and space, if the number of nodes (variables) in the Bayesian network is $n$ and the in-degree (the number of parents) per node is bounded by a constant $d$. Here we present a parallel algorithm capable of computing the exact posterior probabilities for all $n(n-1)$ edges with optimal parallel space efficiency and nearly optimal parallel time efficiency. That is, if $p=2^k$ processors are used, the run-time reduces to $O(5(d+1)n2^{n-k}+k(n-k)^d)$ and the space usage becomes $O(n2^{n-k})$ per processor. Our algorithm is based the observation that the subproblems in the sequential DP algorithm constitute a $n$-$D$ hypercube. We take a delicate way to coordinate the computation of correlated DP procedures such that large amount of data exchange is suppressed. Further, we develop parallel techniques for two variants of the well-known \emph{zeta transform}, which have applications outside the context of Bayesian networks. We demonstrate the capability of our algorithm on datasets with up to 33 variables and its scalability on up to 2048 processors. We apply our algorithm to a biological data set for discovering the yeast pheromone response pathways.

Motivation & Objective

• Address the computational bottleneck of exact Bayesian structure discovery in Bayesian networks, which suffers from exponential time and space complexity.
• Overcome the limitation of sequential dynamic programming algorithms that are constrained by memory usage, especially for networks with more than 25 variables.
• Develop a scalable parallel algorithm that reduces runtime and memory usage per processor while maintaining exact computation of posterior probabilities for all structural features.
• Enable practical discovery of causal relationships in large biological datasets, such as the yeast pheromone response pathway, where structure learning is critical.
• Design a method that suppresses excessive data exchange between parallel processes and efficiently coordinates correlated dynamic programming computations.

Proposed method

• Represents the dynamic programming subproblems of Bayesian structure discovery as an n-dimensional hypercube to exploit inherent parallelism.
• Introduces a novel coordination strategy that minimizes inter-process data transfer during transitions between correlated DP procedures.
• Develops two new parallel techniques for computing variants of the zeta transform, which are fundamental to the algorithm’s efficiency.
• Uses a processor count of $ p = 2^k $ to achieve $ O(5(d+1)n2^{n-k} + k(n-k)^d) $ runtime and $ O(n2^{n-k}) $ space per processor, enabling near-optimal parallel time and optimal space efficiency.
• Applies the algorithm to compute exact posterior probabilities for all $ n(n-1) $ edges in a Bayesian network under a standard (structure-modular) prior, avoiding the limitations of order-modular priors.
• Implements the algorithm using a hybrid approach that combines shared-memory parallelism and efficient memory management to reduce reliance on external storage.

Experimental results

Research questions

• RQ1 Can exact Bayesian structure discovery in Bayesian networks be scaled to larger networks (e.g., 30+ variables) using parallel computing?
• RQ2 How can the exponential space complexity of dynamic programming in structure discovery be effectively reduced through parallelization?
• RQ3 What is the optimal way to coordinate multiple, correlated dynamic programming computations in a parallel setting to minimize communication overhead?
• RQ4 Can the algorithm achieve both near-optimal parallel time and optimal space efficiency simultaneously in exact Bayesian structure discovery?
• RQ5 To what extent can the algorithm recover known biological regulatory pathways from high-dimensional data without prior domain constraints?

Key findings

• The algorithm successfully computed exact posterior probabilities for all 1056 potential edges in a 33-variable biological dataset in 1542 seconds using 2048 cores, setting a new record for exact structure discovery.
• The ParaREBEL algorithm achieved a runtime of $ O(5(d+1)n2^{n-k} + k(n-k)^d) $ with $ p = 2^k $ processors, demonstrating near-optimal parallel time efficiency.
• Memory usage was reduced to $ O(n2^{n-k}) $ per processor, achieving optimal space efficiency, which is critical for scaling beyond 25 variables.
• The resulting network model for the yeast pheromone response pathway, built using edges with posterior probability ≥ 0.1, formed a DAG with 60 edges and showed strong clustering of biologically related variables.
• The network structure closely resembled a known constrained model from prior literature, suggesting that posterior probability-based modeling outperforms greedy or simulated annealing methods in capturing true biological relationships.
• The algorithm demonstrated scalability on up to 2048 processors and is the first practical parallel solution for exact Bayesian structure discovery with provable efficiency guarantees.