[Paper Review] DAGs with NO TEARS: Continuous Optimization for Structure Learning

Xun Zheng, Bryon Aragam+2 more|arXiv (Cornell University)|Mar 4, 2018

Bayesian Modeling and Causal InferenceComputer Science224 citations

TL;DR

The paper reframes DAG structure learning as a continuous optimization problem by introducing a smooth, exact acyclicity constraint (NOTEARS), enabling efficient structure and parameter learning without combinatorial search.

ABSTRACT

Estimating the structure of directed acyclic graphs (DAGs, also known as Bayesian networks) is a challenging problem since the search space of DAGs is combinatorial and scales superexponentially with the number of nodes. Existing approaches rely on various local heuristics for enforcing the acyclicity constraint. In this paper, we introduce a fundamentally different strategy: We formulate the structure learning problem as a purely \emph{continuous} optimization problem over real matrices that avoids this combinatorial constraint entirely. This is achieved by a novel characterization of acyclicity that is not only smooth but also exact. The resulting problem can be efficiently solved by standard numerical algorithms, which also makes implementation effortless. The proposed method outperforms existing ones, without imposing any structural assumptions on the graph such as bounded treewidth or in-degree. Code implementing the proposed algorithm is open-source and publicly available at https://github.com/xunzheng/notears.

Motivation & Objective

• Motivate the NP-hard nature of learning DAGs due to acyclicity and the need for scalable methods.
• Introduce a continuous formulation that replaces discrete DAG constraints with a smooth equality constraint.
• Develop an augmented Lagrangian scheme to optimize the continuous program for joint structure and parameter estimation.
• Demonstrate empirical effectiveness, comparing against state-of-the-art methods and relating to the global minimizer in practice.

Proposed method

• Define F(W) as the regularized LS loss: F(W)= (1/2n)||X - XW||_F^2 + λ||W||_1.
• Characterize acyclicity with a smooth function h(W)=tr(exp(W∘W))−d, where ∘ is the Hadamard product.
• Replace the discrete DAG constraint with the equality h(W)=0, yielding an equality-constrained program (ECP).
• Solve (ECP) via augmented Lagrangian: minimize F(W) + (ρ/2)|h(W)|^2 + α h(W), update α by dual ascent, and iteratively optimize subproblems using L-BFGS or proximal quasi-Newton methods.
• After optimization, apply hard thresholding: Ŵ = W̃_ECP ∘ 1(|W̃_ECP|>ω) to obtain a sparse, acyclic structure.
• Note: The approach leverages standard numerical solvers and can be implemented in ~50 lines of Python.

Experimental results

Research questions

• RQ1 Can a smooth, exact acyclicity constraint replace the combinatorial acyclicity constraint in DAG structure learning?
• RQ2 Do continuous, nonconvex optimization methods with standard solvers yield competitive DAG structure and parameter estimates without restrictive graph assumptions?
• RQ3 How close are solutions from the continuous formulation to the global optimum and to exact DAGs in practice?

Key findings

• NOTEARS achieves state-of-the-art performance without assumptions like bounded treewidth or in-degree.
• The method attains scores comparable to the globally optimal score in practice, though it only guarantees convergence to stationary points.
• Regularization (ℓ1) improves structure recovery in small-sample regimes.
• The approach scales to moderately high dimensions and produces consistent parameter estimates in large samples, with robustness across different noise models.
• The authors provide open-source code implementing NOTEARS at github.com/xunzheng/notears.