[Paper Review] Complexity analysis of Bayesian learning of high-dimensional DAG models and their equivalence classes

Quan Zhou, Hyunwoong Chang|arXiv (Cornell University)|Jan 11, 2021

Bayesian Modeling and Causal Inference59 references4 citations

TL;DR

This paper establishes rapid mixing of a random walk Metropolis-Hastings algorithm for Bayesian structure learning of high-dimensional Gaussian DAG models and their equivalence classes. Under sparsity and high-dimensional assumptions, the mixing time grows polynomially in $n$ and $p$, ensuring efficient exploration of the equivalence class space and strong selection consistency of the empirical Bayes model.

ABSTRACT

Structure learning via MCMC sampling is known to be very challenging because of the enormous search space and the existence of Markov equivalent DAGs. Theoretical results on the mixing behavior are lacking. In this work, we prove the rapid mixing of a random walk Metropolis-Hastings algorithm, which reveals that the complexity of Bayesian learning of sparse equivalence classes grows only polynomially in $n$ and $p$, under some high-dimensional assumptions. A series of high-dimensional consistency results is obtained, including the strong selection consistency of an empirical Bayes model for structure learning. Our proof is based on two new results. First, we derive a general mixing time bound on finite state spaces, which can be applied to local MCMC schemes for other model selection problems. Second, we construct high-probability search paths on the space of equivalence classes with node degree constraints by proving a combinatorial property of DAG comparisons. Simulation studies on the proposed MCMC sampler are conducted to illustrate the main theoretical findings.

Motivation & Objective

• To address the challenge of slow mixing in MCMC samplers for high-dimensional DAG structure learning due to large search spaces and Markov equivalence.
• To establish theoretical guarantees on the mixing behavior of random walk Metropolis-Hastings algorithms on the space of equivalence classes.
• To prove that the complexity of Bayesian learning grows polynomially in sample size $n$ and dimension $p$ under high-dimensional sparsity assumptions.
• To develop a general mixing time bound applicable to local MCMC schemes in model selection problems.
• To construct high-probability search paths on equivalence classes using node degree constraints and a novel combinatorial property of DAG comparisons.

Proposed method

• Derives a general finite-state mixing time bound applicable to local MCMC schemes, enabling analysis of convergence in model selection problems.
• Introduces a canonical path construction on the space of equivalence classes using node degree constraints to ensure high-probability connectivity.
• Employs a random walk Metropolis-Hastings algorithm with local moves on equivalence classes, leveraging local proposals to improve mixing.
• Uses the Poincaré inequality to bound the spectral gap and establish rapid mixing under high-dimensional sparsity.
• Applies a modified empirical Bayes model with a thresholded coefficient structure to define a stable 'true' model under weak faithfulness.
• Validates theoretical findings via simulation studies on the proposed MCMC sampler, demonstrating fast convergence and accurate model recovery.

Experimental results

Research questions

• RQ1 Does the random walk Metropolis-Hastings algorithm on equivalence classes exhibit rapid mixing in high-dimensional settings with $p \gg n$?
• RQ2 Can the mixing time of Bayesian structure learning grow polynomially in $n$ and $p$ under sparsity and high-dimensional assumptions?
• RQ3 Is strong selection consistency achievable for empirical Bayes models in high-dimensional DAG learning when the true model is sparse?
• RQ4 Can a general mixing time bound be derived for local MCMC schemes on finite state spaces in model selection?
• RQ5 How can high-probability search paths be constructed on equivalence classes using combinatorial properties of DAGs and node degree constraints?

Key findings

• The mixing time of the random walk Metropolis-Hastings algorithm on the space of equivalence classes grows polynomially in $n$ and $p$, confirming rapid mixing under high-dimensional sparsity.
• A general finite-state mixing time bound is derived, which applies to local MCMC schemes beyond DAG learning.
• High-probability canonical paths are constructed on equivalence classes using node degree constraints, enabling spectral gap analysis.
• Strong selection consistency is established for the empirical Bayes model under high-dimensional sparsity and weak faithfulness.
• Theoretical results are validated via simulations, showing fast convergence and accurate recovery of sparse DAG structures.
• The method remains effective even when the strong beta-min condition fails, as demonstrated in a non-faithful example with small coefficients.