[Paper Review] Sum-Product Networks: A New Deep Architecture

Hoifung Poon, Pedro Domingos|arXiv (Cornell University)|Feb 14, 2012

Bayesian Modeling and Causal Inference27 references377 citations

TL;DR

This paper introduces Sum-Product Networks (SPNs), a deep architecture composed of sum and product nodes with weighted edges that enable exact and tractable inference by ensuring the network is complete and consistent. SPNs generalize many tractable graphical models and outperform standard deep networks in tasks like image completion, achieving faster and more accurate results due to efficient exact inference via backpropagation and EM-based learning.

ABSTRACT

The key limiting factor in graphical model inference and learning is the complexity of the partition function. We thus ask the question: what are general conditions under which the partition function is tractable? The answer leads to a new kind of deep architecture, which we call sum-product networks (SPNs). SPNs are directed acyclic graphs with variables as leaves, sums and products as internal nodes, and weighted edges. We show that if an SPN is complete and consistent it represents the partition function and all marginals of some graphical model, and give semantics to its nodes. Essentially all tractable graphical models can be cast as SPNs, but SPNs are also strictly more general. We then propose learning algorithms for SPNs, based on backpropagation and EM. Experiments show that inference and learning with SPNs can be both faster and more accurate than with standard deep networks. For example, SPNs perform image completion better than state-of-the-art deep networks for this task. SPNs also have intriguing potential connections to the architecture of the cortex.

Motivation & Objective

• To identify general conditions under which the partition function in graphical models becomes tractable.
• To develop a new deep architecture that supports exact and efficient inference over complex probabilistic models.
• To generalize existing tractable graphical models while enabling scalable learning and inference.
• To propose learning algorithms based on backpropagation and EM for SPNs.
• To demonstrate superior performance of SPNs in inference and learning tasks compared to standard deep networks.

Proposed method

• SPNs are defined as directed acyclic graphs with variables as leaves, sum and product nodes as internal nodes, and weighted edges.
• The network is required to be complete and consistent to ensure it represents a valid probability distribution and enables exact inference.
• Each node in the SPN computes a weighted sum (for sum nodes) or product (for product nodes) of its children’s outputs, with weights interpreted as probabilities.
• The partition function and all marginals are computed in linear time relative to the network size due to the structure and completeness condition.
• Learning is performed via backpropagation and EM, with parameter updates guided by gradient-based optimization.
• The architecture supports exact inference without sampling, enabling fast and accurate probabilistic reasoning.

Experimental results

Research questions

• RQ1 What conditions on a graphical model allow for tractable computation of the partition function and marginals?
• RQ2 Can a deep architecture be designed to support exact inference while remaining expressive and scalable?
• RQ3 How can SPNs be trained efficiently using gradient-based and EM-based methods?
• RQ4 To what extent do SPNs outperform standard deep networks in tasks like image completion?
• RQ5 What connections, if any, exist between SPN structure and neural architecture in the cortex?

Key findings

• SPNs can represent the partition function and all marginals of a graphical model exactly and in linear time when complete and consistent.
• SPNs generalize many tractable graphical models, including Bayesian networks and Markov networks with specific structures.
• SPNs achieve faster and more accurate inference than standard deep networks in image completion tasks.
• The use of backpropagation and EM enables effective learning of SPN parameters with good convergence properties.
• SPNs exhibit intriguing structural similarities to cortical microcircuits, suggesting potential biological plausibility.
• The architecture supports exact inference without approximation, unlike many deep generative models that rely on sampling.