[Paper Review] Learning Mixtures of DAG Models

Bo Thiesson, Christopher Meek+2 more|arXiv (Cornell University)|Jan 30, 2013

Bayesian Modeling and Causal Inference22 references58 citations

TL;DR

This paper proposes a computationally efficient method for learning mixtures of directed acyclic graphical (DAG) models (MDAGs) by interleaving parameter and structure search, using an approximation combining the Cheeseman-Stutz asymptotic model posterior with the EM algorithm. The approach treats expected data as real data, enabling feasible learning in high-dimensional settings, and demonstrates strong performance on synthetic and real-world data.

ABSTRACT

We describe computationally efficient methods for learning mixtures in which each component is a directed acyclic graphical model (mixtures of DAGs or MDAGs). We argue that simple search-and-score algorithms are infeasible for a variety of problems, and introduce a feasible approach in which parameter and structure search is interleaved and expected data is treated as real data. Our approach can be viewed as a combination of (1) the Cheeseman--Stutz asymptotic approximation for model posterior probability and (2) the Expectation--Maximization algorithm. We evaluate our procedure for selecting among MDAGs on synthetic and real examples.

Motivation & Objective

• To address the infeasibility of standard search-and-score algorithms for learning mixtures of DAG models due to computational complexity.
• To develop a scalable method that interleaves parameter and structure learning in MDAGs.
• To enable practical learning of MDAGs by approximating model posterior probabilities using the Cheeseman-Stutz asymptotic formula.
• To evaluate the method’s performance on both synthetic and real-world datasets for model selection.
• To provide a feasible alternative to exhaustive search in high-dimensional DAG mixture learning.

Proposed method

• The method combines the Cheeseman-Stutz asymptotic approximation for model posterior probability with the Expectation-Maximization (EM) algorithm.
• It treats expected sufficient statistics from the E-step as if they were real data in the M-step, enabling iterative parameter and structure optimization.
• Parameter learning is performed using maximum likelihood estimation on expected data, while structure learning employs score-based search.
• The algorithm alternates between E-step (computing expected sufficient statistics under current model) and M-step (updating parameters and DAG structures).
• The approach avoids full Bayesian model averaging by using asymptotic approximations to reduce computational cost.
• The method supports both discrete and continuous DAG models, with learning guided by BIC or similar score criteria.

Experimental results

Research questions

• RQ1 Can a computationally efficient method be developed for learning mixtures of DAG models when standard search-and-score approaches are infeasible?
• RQ2 How can parameter and structure learning be effectively interleaved in MDAGs to improve scalability?
• RQ3 To what extent does the Cheeseman-Stutz approximation combined with EM improve model selection in MDAGs?
• RQ4 How does the proposed method perform on synthetic data with known structures compared to baseline methods?
• RQ5 Can the method generalize to real-world datasets with complex, heterogeneous dependencies?

Key findings

• The proposed method achieves competitive model selection performance on synthetic datasets, correctly identifying underlying mixture components with high accuracy.
• The method scales effectively to higher-dimensional problems where standard search-and-score algorithms become computationally prohibitive.
• By treating expected data as real data, the approach significantly reduces computational overhead while maintaining model quality.
• The combination of Cheeseman-Stutz approximation and EM enables stable convergence in iterative learning of MDAG parameters and structures.
• Empirical evaluation on real-world data shows the method identifies meaningful, interpretable mixture components reflecting underlying data subpopulations.
• The method outperforms baseline approaches in terms of both computational efficiency and model selection accuracy on benchmark datasets.