[论文解读] DAGs with NO TEARS: Continuous Optimization for Structure Learning

Xun Zheng, Bryon Aragam+2 more|arXiv (Cornell University)|Mar 4, 2018

Bayesian Modeling and Causal Inference被引用 224

一句话总结

本文通过引入一个平滑且精确的无环约束（NOTEARS），将 DAG 结构学习重新表述为一个连续优化问题，从而在无需组合搜索的情况下实现高效的结构和参数学习。

ABSTRACT

Estimating the structure of directed acyclic graphs (DAGs, also known as Bayesian networks) is a challenging problem since the search space of DAGs is combinatorial and scales superexponentially with the number of nodes. Existing approaches rely on various local heuristics for enforcing the acyclicity constraint. In this paper, we introduce a fundamentally different strategy: We formulate the structure learning problem as a purely \emph{continuous} optimization problem over real matrices that avoids this combinatorial constraint entirely. This is achieved by a novel characterization of acyclicity that is not only smooth but also exact. The resulting problem can be efficiently solved by standard numerical algorithms, which also makes implementation effortless. The proposed method outperforms existing ones, without imposing any structural assumptions on the graph such as bounded treewidth or in-degree. Code implementing the proposed algorithm is open-source and publicly available at https://github.com/xunzheng/notears.

研究动机与目标

• Motivate the NP-hard nature of learning DAGs due to acyclicity and the need for scalable methods.
• Introduce a continuous formulation that replaces discrete DAG constraints with a smooth equality constraint.
• Develop an augmented Lagrangian scheme to optimize the continuous program for joint structure and parameter estimation.
• Demonstrate empirical effectiveness, comparing against state-of-the-art methods and relating to the global minimizer in practice.

提出的方法

• Define F(W) as the regularized LS loss: F(W)= (1/2n)||X - XW||_F^2 + λ||W||_1.
• Characterize acyclicity with a smooth function h(W)=tr(exp(W∘W))−d, where ∘ is the Hadamard product.
• Replace the discrete DAG constraint with the equality h(W)=0, yielding an equality-constrained program (ECP).
• Solve (ECP) via augmented Lagrangian: minimize F(W) + (ρ/2)|h(W)|^2 + α h(W), update α by dual ascent, and iteratively optimize subproblems using L-BFGS or proximal quasi-Newton methods.
• After optimization, apply hard thresholding: Ŵ = W̃_ECP ∘ 1(|W̃_ECP|>ω) to obtain a sparse, acyclic structure.
• Note: The approach leverages standard numerical solvers and can be implemented in ~50 lines of Python.]

实验结果

研究问题

• RQ1 Can a smooth, exact acyclicity constraint replace the combinatorial acyclicity constraint in DAG structure learning?
• RQ2 Do continuous, nonconvex optimization methods with standard solvers yield competitive DAG structure and parameter estimates without restrictive graph assumptions?
• RQ3 How close are solutions from the continuous formulation to the global optimum and to exact DAGs in practice?

主要发现

F(W)	F(W_G)	F(Ŵ)	F(W̃_ECP)	Δ(W_G, Ŵ)	||Ŵ−W_G||	||W−W_G||
5.11	3.85	5.36	3.88	-1.52	0.07	3.38
16.04	12.81	13.49	12.90	-0.68	0.12	3.15
4.99	4.97	5.02	4.95	-0.05	0.02	0.40
15.93	13.32	14.03	13.46	-0.71	0.12	2.95
4.99	3.77	4.70	3.85	-0.93	0.08	3.31
23.33	16.19	17.31	16.69	-1.12	0.15	5.08
4.96	4.94	5.05	4.99	-0.11	0.04	0.29
23.29	17.56	19.70	18.43	-2.13	0.13	4.34

• NOTEARS achieves state-of-the-art performance without assumptions like bounded treewidth or in-degree.
• The method attains scores comparable to the globally optimal score in practice, though it only guarantees convergence to stationary points.
• Regularization (ℓ1) improves structure recovery in small-sample regimes.
• The approach scales to moderately high dimensions and produces consistent parameter estimates in large samples, with robustness across different noise models.
• The authors provide open-source code implementing NOTEARS at github.com/xunzheng/notears.