[Paper Review] On the Role of Sparsity and DAG Constraints for Learning Linear DAGs

Ignavier Ng, AmirEmad Ghassami+1 more|arXiv (Cornell University)|Jun 17, 2020

Bayesian Modeling and Causal Inference58 references57 citations

TL;DR

The paper proposes GOLEM, a likelihood-based DAG learning method using soft sparsity and DAG constraints to avoid hard acyclicity constraints, and demonstrates its effectiveness for linear DAGs in identifiable and nonidentifiable settings with scalable GPU-based optimization.

ABSTRACT

Learning graphical structures based on Directed Acyclic Graphs (DAGs) is a challenging problem, partly owing to the large search space of possible graphs. A recent line of work formulates the structure learning problem as a continuous constrained optimization task using the least squares objective and an algebraic characterization of DAGs. However, the formulation requires a hard DAG constraint and may lead to optimization difficulties. In this paper, we study the asymptotic role of the sparsity and DAG constraints for learning DAG models in the linear Gaussian and non-Gaussian cases, and investigate their usefulness in the finite sample regime. Based on the theoretical results, we formulate a likelihood-based score function, and show that one only has to apply soft sparsity and DAG constraints to learn a DAG equivalent to the ground truth DAG. This leads to an unconstrained optimization problem that is much easier to solve. Using gradient-based optimization and GPU acceleration, our procedure can easily handle thousands of nodes while retaining a high accuracy. Extensive experiments validate the effectiveness of our proposed method and show that the DAG-penalized likelihood objective is indeed favorable over the least squares one with the hard DAG constraint.

Motivation & Objective

• Assess the necessity of hard DAG constraints and sparsity for learning linear DAGs under various model assumptions.
• Develop a continuous unconstrained optimization framework (GOLEM) based on likelihood with soft constraints.
• Compare likelihood-based GOLEM to regression-based LS methods and to NOTEARS across identifiable and nonidentifiable settings.
• Demonstrate scalability to thousands of nodes using GPU acceleration and provide empirical validation on synthetic and real data.

Proposed method

• Formulate a likelihood-based score with soft sparsity and DAG constraints, leading to an unconstrained optimization problem for B, the weighted adjacency matrix.
• Use two likelihood objectives, L1 (likelihood with non-equal variances) and L2 (likelihood with equal variances), and add an l1 penalty plus a DAG penalty h(B) = tr(exp(B ∘ B)) − d.
• Solve the nonconvex optimization with gradient-based methods (Adam) and initialize GOLEM-NV from GOLEM-EV to avoid poor local minima.
• Apply a post-processing threshold to edge weights to obtain a DAG, then iteratively remove cycles if needed to produce a DAG.
• Analyze theoretical roles of sparsity and DAG constraints under general Gaussian and identifiable regimes, showing soft constraints suffice asymptotically.

Experimental results

Research questions

• RQ1 Does a hard DAG constraint improve graph recovery compared to soft, likelihood-based constraints in linear DAG learning?
• RQ2 Under which model assumptions (Gaussian with equal/non-equal variances, non-Gaussian, LiNGAM identifiability) do sparsity and DAG constraints ensure recovery of a DAG equivalent to the ground truth?
• RQ3 How does a likelihood-based GOLEM compare to LS-based and NOTEARS approaches in identifiable and nonidentifiable settings in terms of accuracy and scalability?
• RQ4 Can GOLEM scale to thousands of nodes with GPU acceleration while maintaining high accuracy across graph types (ER and SF) and sample sizes?
• RQ5 What is the impact of weight scaling and finite-sample effects on the performance of GOLEM versus baselines?

Key findings

• GOLEM with soft sparsity and DAG constraints asymptotically recovers a DAG-equivalent to the ground truth under mild assumptions.
• In Gaussian-EV settings, sparsity alone can suffice to recover the ground-truth DAG; in Gaussian-NV settings, DAG regularization becomes necessary for reliable recovery.
• GOLEM-NV and GOLEM-EV outperform baseline methods, especially on larger and denser graphs, and can handle thousands of nodes with GPU acceleration.
• In identifiable cases (LiNGAM-like and equal-noise Gaussian), gradient-based GOLEM variants outperform or match competitive baselines across noise types (Gaussian, Exponential, Gumbel).
• GOLEM shows scalability advantages over NOTEARS-L1, maintaining accuracy as graph size grows, aided by GPU-accelerated cubic-time computations.
• Thresholding post-processing effectively yields acyclic graphs, leveraging the DAG penalty to prune cycles in finite samples.