[Paper Review] Learning Arithmetic Circuits

Daniel Lowd, Pedro Domingos|arXiv (Cornell University)|Jun 13, 2012

Bayesian Modeling and Causal InferenceComputer Science19 references70 citations

TL;DR

This paper proposes learning arithmetic circuits with a penalty on circuit size to directly optimize for efficient inference, combining Bayesian network learning with circuit compilation. By using circuit size as a proxy for inference cost, the method learns tractable models with high treewidth that outperform standard context-specific Bayesian networks in accuracy and speed.

ABSTRACT

Graphical models are usually learned without regard to the cost of doing inference with them. As a result, even if a good model is learned, it may perform poorly at prediction, because it requires approximate inference. We propose an alternative: learning models with a score function that directly penalizes the cost of inference. Specifically, we learn arithmetic circuits with a penalty on the number of edges in the circuit (in which the cost of inference is linear). Our algorithm is equivalent to learning a Bayesian network with context-specific independence by greedily splitting conditional distributions, at each step scoring the candidates by compiling the resulting network into an arithmetic circuit, and using its size as the penalty. We show how this can be done efficiently, without compiling a circuit from scratch for each candidate. Experiments on several real-world domains show that our algorithm is able to learn tractable models with very large treewidth, and yields more accurate predictions than a standard context-specific Bayesian network learner, in far less time.

Motivation & Objective

• To address the inefficiency of inference in learned graphical models, even when the models are accurate.
• To develop a learning framework that directly optimizes for tractable inference by penalizing model complexity.
• To integrate circuit compilation into the learning process to guide structure selection.
• To enable scalable learning of high-treewidth models that remain tractable for inference.
• To outperform standard context-specific Bayesian network learners in both accuracy and runtime.

Proposed method

• The method uses a score function that combines model likelihood with a penalty on the number of edges in the arithmetic circuit.
• At each learning step, candidate splits in conditional distributions are evaluated by compiling the resulting Bayesian network into an arithmetic circuit.
• The size of the compiled circuit is used as a proxy for inference cost, guiding greedy structure search.
• The algorithm avoids full recompilation for each candidate by incrementally updating the circuit structure.
• The approach leverages the fact that inference cost in arithmetic circuits is linear in the number of edges.
• It combines greedy Bayesian network learning with circuit compilation to balance model fit and tractability.

Experimental results

Research questions

• RQ1 Can we learn graphical models that are both accurate and efficient to infer?
• RQ2 How can we incorporate inference cost directly into the model learning process?
• RQ3 Can we use circuit size as an effective proxy for inference complexity in structure learning?
• RQ4 Does integrating circuit compilation into learning lead to better performance than standard Bayesian network learners?
• RQ5 Can we scale learning to high-treewidth models while maintaining tractability?

Key findings

• The proposed method learns arithmetic circuits with significantly higher treewidth than standard context-specific Bayesian network learners.
• The learned models achieve higher predictive accuracy than baseline Bayesian network learners on real-world datasets.
• The method runs orders of magnitude faster than standard context-specific Bayesian network learning algorithms.
• The use of circuit size as a penalty effectively balances model fit and inference efficiency.
• The incremental compilation technique enables efficient evaluation of candidate structures without full recompilation.
• Empirical results confirm that the method scales well and produces tractable models suitable for real-world applications.