[Paper Review] BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery

Chris Cundy, Aditya Grover+1 more|arXiv (Cornell University)|Dec 6, 2021

Bayesian Modeling and Causal Inference4 citations

TL;DR

BCD Nets proposes a scalable variational inference framework for Bayesian causal discovery in linear-Gaussian structural equation models, using deep neural networks to model posterior distributions over DAGs with continuous relaxations and horseshoe priors. It significantly outperforms maximum-likelihood methods on structural Hamming distance in low-data regimes by quantifying uncertainty in DAG structure inference.

ABSTRACT

A structural equation model (SEM) is an effective framework to reason over causal relationships represented via a directed acyclic graph (DAG). Recent advances have enabled effective maximum-likelihood point estimation of DAGs from observational data. However, a point estimate may not accurately capture the uncertainty in inferring the underlying graph in practical scenarios, wherein the true DAG is non-identifiable and/or the observed dataset is limited. We propose Bayesian Causal Discovery Nets (BCD Nets), a variational inference framework for estimating a distribution over DAGs characterizing a linear-Gaussian SEM. Developing a full Bayesian posterior over DAGs is challenging due to the the discrete and combinatorial nature of graphs. We analyse key design choices for scalable VI over DAGs, such as 1) the parametrization of DAGs via an expressive variational family, 2) a continuous relaxation that enables low-variance stochastic optimization, and 3) suitable priors over the latent variables. We provide a series of experiments on real and synthetic data showing that BCD Nets outperform maximum-likelihood methods on standard causal discovery metrics such as structural Hamming distance in low data regimes.

Motivation & Objective

• To address the limitations of point-estimate methods in causal discovery, which fail to capture uncertainty in DAG structure under finite or non-identifiable data.
• To develop a scalable Bayesian framework for estimating a posterior distribution over DAGs in linear-Gaussian SEMs.
• To enable uncertainty-aware causal inference in high-stakes, low-data applications such as medicine and policy.
• To overcome the computational intractability of full Bayesian inference over discrete, combinatorial DAGs using continuous relaxations and amortized inference.

Proposed method

• BCD Nets uses a deep neural network to parameterize a factorized variational posterior over SEM parameters (edge weights and noise variances), exploiting DAG decomposition into triangular matrices and permutations.
• It employs a continuous relaxation of permutations via the Gumbel-Softmax trick to enable low-variance stochastic optimization of the variational objective.
• A horseshoe prior is applied to edge weights to promote sparsity and improve structural generalization.
• The method uses amortized inference with a neural network to efficiently approximate the posterior over DAGs, avoiding exhaustive enumeration.
• It formulates the variational objective as a differentiable lower bound on the marginal likelihood, enabling end-to-end training with gradient-based optimization.
• The framework supports both equal and non-equal noise variance settings in linear-Gaussian SEMs, with flexible modeling of the joint posterior over DAGs and parameters.

Experimental results

Research questions

• RQ1 Can a scalable variational inference framework effectively quantify uncertainty in DAG structure for linear-Gaussian SEMs?
• RQ2 How does Bayesian estimation with proper priors compare to maximum-likelihood point estimation in low-data regimes?
• RQ3 What design choices—such as parametrization, relaxation, and prior selection—most significantly impact performance and scalability in Bayesian causal discovery?
• RQ4 To what extent does the use of continuous relaxations and amortized inference maintain accuracy while enabling efficient optimization over combinatorial DAG spaces?
• RQ5 How robust is the method to model misspecification, such as non-linear relationships or unobserved confounders?

Key findings

• BCD Nets outperforms maximum-likelihood baselines on structural Hamming distance (SHD) in low-data regimes, with SHD dropping from ~30 to ~11 on 32-node Erdős-Rényi graphs.
• The method achieves a 350-minute average training time on 64-node graphs, outperforming GADGET (635 minutes) and GOLEM (65 minutes) in scalability and accuracy trade-offs.
• Ablation studies show that removing the horseshoe prior or using a fixed Sinkhorn step count increases SHD from 11 to around 30, indicating critical importance of sparsity and adaptive relaxation.
• The KL-divergence between sample and induced covariance matrices increases only modestly under Laplace prior, but SHD rises sharply, indicating poor structural generalization despite data fit.
• As dataset size increases, the performance gap between BCD Nets and MLE methods narrows, confirming the method’s advantage is most pronounced in low-data regimes.
• The framework is robust to model misspecification in terms of optimization, but assumes no unobserved confounders and linear relationships, which may limit real-world applicability if violated.