[Paper Review] Learning Bayesian Network Structure from Massive Datasets: The "Sparse Candidate" Algorithm

Nir Friedman, Iftach Nachman+1 more|arXiv (Cornell University)|Jan 1, 1999

Bayesian Modeling and Causal Inference23 references519 citations

TL;DR

This paper introduces the 'Sparse Candidate' algorithm to accelerate Bayesian network structure learning from massive datasets by iteratively restricting the set of candidate parents for each variable to a small, data-driven subset. By combining statistical cues (like mutual information) with iterative refinement using learned network structures, the method achieves significant speedups—up to 3x faster than greedy hill-climbing—while maintaining or improving score quality, especially on high-dimensional data with thousands of attributes.

ABSTRACT

Learning Bayesian networks is often cast as an optimization problem, where the computational task is to find a structure that maximizes a statistically motivated score. By and large, existing learning tools address this optimization problem using standard heuristic search techniques. Since the search space is extremely large, such search procedures can spend most of the time examining candidates that are extremely unreasonable. This problem becomes critical when we deal with data sets that are large either in the number of instances, or the number of attributes. In this paper, we introduce an algorithm that achieves faster learning by restricting the search space. This iterative algorithm restricts the parents of each variable to belong to a small subset of candidates. We then search for a network that satisfies these constraints. The learned network is then used for selecting better candidates for the next iteration. We evaluate this algorithm both on synthetic and real-life data. Our results show that it is significantly faster than alternative search procedures without loss of quality in the learned structures.

Motivation & Objective

• To address the computational infeasibility of exhaustive search in large Bayesian network structure learning.
• To reduce search space by restricting candidate parents per variable using statistical dependencies.
• To improve search efficiency without sacrificing network quality on large-scale datasets.
• To enable scalable learning in high-dimensional domains (e.g., gene expression, text) where standard methods fail due to memory and time constraints.

Proposed method

• Uses mutual information between variables as a statistical cue to pre-select a small set of candidate parents for each variable.
• Applies an iterative process: learn a network under current candidate constraints, then refine the candidate sets using the learned structure.
• Employs a score-based heuristic (e.g., BIC or BDe) to guide candidate selection in each iteration.
• Restricts search to O(kn) candidates per variable, where k << n, instead of O(n²), drastically reducing search space.
• Uses the learned network to re-estimate dependencies and improve candidate sets in subsequent iterations.
• Combines with standard heuristic search (e.g., hill-climbing) under the constrained parent sets to maximize score efficiently.

Experimental results

Research questions

• RQ1 Can restricting the parent search space using statistical dependencies significantly reduce learning time without degrading network quality?
• RQ2 How effective is iterative refinement of candidate parents using the learned network structure?
• RQ3 Can the method scale to datasets with thousands of attributes where standard methods fail?
• RQ4 Does the use of mutual information as a pruning heuristic lead to better convergence than random or uniform candidate selection?
• RQ5 Can theoretical guarantees on complexity be achieved under the sparse candidate constraint?

Key findings

• On a 100-attribute text dataset, the Sparse Candidate algorithm achieved networks with comparable scores to greedy hill-climbing in half the time and with half the number of sufficient statistics.
• On a 200-attribute text dataset, the speedup exceeded 3x compared to greedy hill-climbing.
• In a high-dimensional gene expression dataset (800 genes), greedy hill-climbing failed due to memory constraints, while the Sparse Candidate method successfully learned high-scoring networks.
• The first iteration already produced reasonably high-scoring networks, and subsequent iterations further improved the score, demonstrating the value of iterative refinement.
• The discrepancy measure (based on learned structure) showed a slower learning curve than the score measure, indicating that score-based candidate selection is more effective.
• The method enables learning in domains with thousands of attributes where standard approaches are infeasible, as demonstrated in ongoing work on real gene expression data.