[Paper Review] Information-theoretic limits of Bayesian network structure learning

Asish Ghoshal, Jean Honorio|arXiv (Cornell University)|Jan 27, 2016

Bayesian Modeling and Causal Inference3 citations

TL;DR

This paper establishes information-theoretic lower bounds on the sample complexity required to recover the structure of Bayesian networks (BNs), showing that Ω(m) samples are needed for non-sparse BNs and Ω(k log m + k²/m) for sparse BNs with maximum in-degree k. It introduces a general recipe based on an extended Fano’s inequality applicable to exponential family BNs, and derives tight bounds for CPT-based, Gaussian, noisy-OR, and logistic regression BNs, revealing that existing algorithms like SparsityBoost are far from optimal in sample efficiency.

ABSTRACT

In this paper, we study the information-theoretic limits of learning the structure of Bayesian networks (BNs), on discrete as well as continuous random variables, from a finite number of samples. We show that the minimum number of samples required by any procedure to recover the correct structure grows as $Ω(m)$ and $Ω(k \log m + (k^2/m))$ for non-sparse and sparse BNs respectively, where $m$ is the number of variables and $k$ is the maximum number of parents per node. We provide a simple recipe, based on an extension of the Fano's inequality, to obtain information-theoretic limits of structure recovery for any exponential family BN. We instantiate our result for specific conditional distributions in the exponential family to characterize the fundamental limits of learning various commonly used BNs, such as conditional probability table based networks, gaussian BNs, noisy-OR networks, and logistic regression networks. En route to obtaining our main results, we obtain tight bounds on the number of sparse and non-sparse essential-DAGs. Finally, as a byproduct, we recover the information-theoretic limits of sparse variable selection for logistic regression.

Motivation & Objective

• To determine the fundamental minimum sample size required for consistent recovery of Bayesian network (BN) structure from data.
• To develop a general method applicable to any exponential family BN to derive information-theoretic lower bounds on sample complexity.
• To characterize the sample complexity limits for specific BN types, including conditional probability table (CPT), Gaussian, noisy-OR, and logistic regression networks.
• To quantify the gap between current algorithms (e.g., SparsityBoost) and information-theoretic limits in terms of sample efficiency.
• To extend Fano’s inequality to handle latent variables and derive bounds on KL divergence between exponential family distributions.

Proposed method

• Derives a general recipe for information-theoretic limits using an extension of Fano’s inequality to settings with latent variables.
• Applies the extended Fano’s inequality to exponential family distributions, leveraging the natural parameterization and sufficient statistics.
• Establishes tight bounds on the number of essential DAGs (both sparse and non-sparse) to quantify the complexity of structure recovery.
• Computes mutual information upper bounds between observed data and the true DAG structure for each BN type, using parametric assumptions.
• Uses the mutual information bounds in conjunction with Fano’s inequality to derive lower bounds on the required sample size for reliable structure recovery.
• Applies the framework to specific BN models: CPT-based, Gaussian, noisy-OR, and logistic regression BNs, deriving explicit sample complexity lower bounds for each.

Experimental results

Research questions

• RQ1 What is the minimum number of samples required to reliably recover the true DAG structure of a Bayesian network, regardless of the learning algorithm?
• RQ2 How do the information-theoretic sample complexity limits differ between non-sparse and sparse Bayesian networks?
• RQ3 To what extent do existing structure learning algorithms, such as SparsityBoost, deviate from the fundamental sample complexity limits?
• RQ4 Can a unified framework be developed to derive information-theoretic limits for any Bayesian network within the exponential family?
• RQ5 What is the impact of model-specific parameters (e.g., minimum probability θ_min or weight norm w_max^1) on the fundamental sample complexity?

Key findings

• The minimum number of samples required to recover the structure of a non-sparse BN is Ω(m), where m is the number of variables.
• For sparse BNs with maximum in-degree k, the required sample size is Ω(k log m + k²/m), which matches the upper bound of O(k² log m) in the regime where k is small relative to m.
• The sample complexity lower bound for logistic regression BNs scales as Ω(k² log m) when the ℓ₁ norm of the weight vectors is bounded by w_max^1 ≤ 1/k, matching the upper bound from Ravikumar et al. [3], proving information-theoretic optimality.
• The SparsityBoost algorithm for binary CPT BNs has a sample complexity of O(m² / θ_min), which is significantly worse than the information-theoretic lower bound of Ω((k log m + k²/m) / log(1/θ_min)).
• Learning layered Bayesian networks—where node ordering is known up to layers—is as hard as learning general BNs in terms of sample complexity, indicating that layering does not reduce the fundamental difficulty.
• The paper provides a new upper bound on the KL divergence between two exponential family distributions, expressed as the inner product of the difference in natural parameters and the expected sufficient statistics, which is a key technical contribution.