[Paper Review] Estimating high-dimensional directed acyclic graphs with the PC-algorithm

Markus Kalisch, Peter Bühlmann|ArXiv.org|Oct 20, 2005

Bayesian Modeling and Causal InferenceComputer Science25 references611 citations

TL;DR

This paper establishes the theoretical consistency of the PC-algorithm for estimating high-dimensional directed acyclic graphs (DAGs) under minimal sparsity assumptions, proving it consistently recovers the true DAG skeleton even when the number of variables $ p $ grows faster than any polynomial in sample size $ n $. The method leverages conditional independence testing via partial correlations and is shown to be robust to tuning parameter choice in practice.

ABSTRACT

We consider the PC-algorithm Spirtes et. al. (2000) for estimating the skeleton of a very high-dimensional acyclic directed graph (DAG) with corresponding Gaussian distribution. The PC-algorithm is computationally feasible for sparse problems with many nodes, i.e. variables, and it has the attractive property to automatically achieve high computational efficiency as a function of sparseness of the true underlying DAG. We prove consistency of the algorithm for very high-dimensional, sparse DAGs where the number of nodes is allowed to quickly grow with sample size n, as fast as O(n^a) for any 0

Motivation & Objective

• To establish theoretical consistency of the PC-algorithm for estimating the skeleton of high-dimensional DAGs where the number of variables $ p $ grows rapidly with sample size $ n $.
• To demonstrate that the PC-algorithm remains computationally feasible and statistically consistent under minimal sparsity assumptions, requiring only that neighborhood sizes are of lower order than $ n $.
• To show that the algorithm is robust to the choice of its single tuning parameter (significance level) in finite samples, based on empirical evidence.
• To provide a rigorous asymptotic justification for the use of the PC-algorithm in high-dimensional settings where $ p > n $, filling a gap in existing methods for DAG structure learning.

Proposed method

• The PC-algorithm is applied to estimate the skeleton of a DAG by iteratively removing edges based on conditional independence tests using partial correlations.
• The algorithm starts from a complete undirected graph and proceeds in levels, testing conditional independence given increasing sets of conditioning variables up to a maximum level $ m_n $.
• Conditional independence is assessed using a significance level $ \alpha_n = 2(1 - \Phi(n^{1/2}c_n/2)) $, which shrinks with sample size to ensure consistency.
• The method relies on the faithfulness assumption and the d-separation criterion to link conditional independence in the data to graph structure.
• Theoretical analysis uses concentration inequalities and tail bounds on the sampling distribution of partial correlations to control type I and II errors.
• The proof establishes that the probability of error in edge selection vanishes as $ n \to \infty $, even when $ p = O(n^a) $ for any $ 0 < a < \infty $, under mild sparsity conditions.

Experimental results

Research questions

• RQ1 Can the PC-algorithm consistently estimate the skeleton of a high-dimensional DAG when the number of variables $ p $ grows faster than any polynomial in sample size $ n $?
• RQ2 Does the PC-algorithm maintain consistency under minimal sparsity assumptions, where the neighborhood size of each node is of lower order than $ n $?
• RQ3 How sensitive is the PC-algorithm to the choice of its single tuning parameter (significance level) in finite samples?
• RQ4 Under what conditions does the PC-algorithm achieve consistent DAG skeleton recovery in high-dimensional settings with $ p > n $?
• RQ5 Is the PC-algorithm computationally feasible and statistically valid in high-dimensional sparse DAGs, even when $ p $ grows super-polynomially with $ n $?

Key findings

• The PC-algorithm is provably consistent for estimating the skeleton of a high-dimensional DAG when $ p = O(n^a) $ for any $ 0 < a < \infty $, as $ n \to \infty $, under minimal sparsity assumptions.
• The algorithm maintains computational feasibility and statistical consistency in sparse DAGs, even when $ p $ grows faster than any polynomial in $ n $, due to its adaptive sparsity-aware structure.
• The probability of error in edge selection decays exponentially fast, specifically bounded by $ O(\exp(-C_5(n - m_n)c_n^2)) $, which vanishes as $ n \to \infty $.
• The method is robust to the choice of the significance level $ \alpha_n $, as empirical results show consistent performance across different values of the tuning parameter.
• The PC-algorithm achieves consistency even when the true DAG has neighborhoods of size growing slower than $ n $, making it applicable to a wide class of high-dimensional models.
• Theoretical results confirm that the sample PC-algorithm coincides with the population version with high probability, provided the stopping level $ m_n $ is chosen such that $ m_n = O(n^{1-b}) $, ensuring convergence.