[Paper Review] Learning Discrete Bayesian Networks from Continuous Data

Yi‐Chun Chen, Tim A. Wheeler+1 more|arXiv (Cornell University)|Dec 8, 2015

Bayesian Modeling and Causal Inference24 references9 citations

TL;DR

This paper proposes a principled Bayesian discretization method for learning discrete Bayesian networks from continuous data, using a quadratic-time dynamic programming approach that improves upon the cubic complexity of traditional minimum description length (MDL) methods. The method simultaneously learns network structure and optimal discretization, achieving superior likelihood and accuracy on benchmark datasets like Auto MPG and Iris, especially in structure-aware scenarios where marginal methods fail.

ABSTRACT

Learning Bayesian networks from raw data can help provide insights into the relationships between variables. While real data often contains a mixture of discrete and continuous-valued variables, many Bayesian network structure learning algorithms assume all random variables are discrete. Thus, continuous variables are often discretized when learning a Bayesian network. However, the choice of discretization policy has significant impact on the accuracy, speed, and interpretability of the resulting models. This paper introduces a principled Bayesian discretization method for continuous variables in Bayesian networks with quadratic complexity instead of the cubic complexity of other standard techniques. Empirical demonstrations show that the proposed method is superior to the established minimum description length algorithm. In addition, this paper shows how to incorporate existing methods into the structure learning process to discretize all continuous variables and simultaneously learn Bayesian network structures.

Motivation & Objective

• To address the challenge of learning discrete Bayesian networks from mixed discrete-continuous data, where continuous variables are typically discretized with suboptimal policies.
• To develop a structure-aware discretization method that jointly optimizes network structure and discretization thresholds, rather than treating them sequentially.
• To overcome the cubic time complexity of existing MDL-based discretization methods while improving model likelihood and predictive accuracy.
• To demonstrate that incorporating variable dependencies during discretization leads to better joint probability representations than marginal-based or heuristic methods.

Proposed method

• Proposes a Bayesian discretization framework that uses a prior over discretization boundaries and computes posterior probabilities using dynamic programming with O(n²) complexity, where n is the number of data points.
• Integrates discretization into the structure learning pipeline via an alternating optimization loop: first learn the network structure using discrete data, then refine discretization thresholds based on the current structure.
• Uses a non-informative prior on discretization edges and computes the posterior probability of each candidate split using a likelihood model that accounts for conditional dependencies in the network.
• Applies dynamic programming to efficiently search over all possible discretization boundaries for each continuous variable, minimizing the description length of the joint model.
• Employs a reversible jump Markov chain Monte Carlo (RJ-MCMC) or greedy search to explore the space of network structures and discretization policies in tandem.
• Validates the method on synthetic and real-world datasets, comparing against MDL and marginal discretization (e.g., Bayesian Blocks) using log-likelihood and prediction accuracy.

Experimental results

Research questions

• RQ1 Can a Bayesian discretization method that accounts for conditional dependencies in the network structure outperform marginal-based or MDL-based discretization in terms of model likelihood and accuracy?
• RQ2 Does a joint learning approach for structure and discretization lead to better performance than sequential or iterative methods?
• RQ3 Is it possible to reduce the computational complexity of optimal discretization from cubic to quadratic time while maintaining or improving model quality?
• RQ4 How does the proposed method compare to MDL and marginal methods in capturing true underlying conditional distributions in structure-aware settings?

Key findings

• The proposed Bayesian discretization method achieves significantly better log-likelihood on the Auto MPG dataset (−25.94 vs. −26.83 for MDL), indicating a superior joint model fit.
• On the Iris dataset with a naive Bayes structure, both the Bayesian method and MDL produced identical discretization edges and 94% prediction accuracy, but the Bayesian method achieved a higher mean log-likelihood (−2.25 vs. −2.50).
• In a synthetic A→B network with conditional uniform distributions, the Bayesian method correctly identified the true discretization edge at 1.0, while marginal methods like Bayesian Blocks failed and produced a worse log-likelihood (−0.69).
• The method reduces discretization complexity from O(n³) to O(n²) by using an efficient dynamic programming formulation, enabling faster computation on large datasets.
• The joint learning of structure and discretization leads to more accurate and interpretable models, especially in cases where variable dependencies are critical for correct discretization.