[Paper Review] Probabilistic Deep Learning using Random Sum-Product Networks

Robert Peharz, Antonio Vergari+5 more|arXiv (Cornell University)|Sep 5, 2018

Adversarial Robustness in Machine Learning31 references13 citations

TL;DR

This paper introduces RAT-SPNs, a novel probabilistic deep learning framework that combines the efficiency and interpretability of sum-product networks (SPNs) with the training flexibility of deep learning. By using random SPN structures trained via stochastic gradient descent and automatic differentiation, RAT-SPNs achieve deep learning-level performance while maintaining exact likelihood computation, well-calibrated uncertainty, and robustness to missing features and outliers.

ABSTRACT

The need for consistent treatment of uncertainty has recently triggered increased interest in probabilistic deep learning methods. However, most current approaches have severe limitations when it comes to inference, since many of these models do not even permit to evaluate exact data likelihoods. Sum-product networks (SPNs), on the other hand, are an excellent architecture in that regard, as they allow to efficiently evaluate likelihoods, as well as arbitrary marginalization and conditioning tasks. Nevertheless, SPNs have not been fully explored as serious deep learning models, likely due to their special structural requirements, which complicate learning. In this paper, we make a drastic simplification and use random SPN structures which are trained in a classical deep learning manner, i.e. employing automatic differentiation, SGD, and GPU support. The resulting models, called RAT-SPNs, yield prediction results comparable to deep neural networks, while still being interpretable as generative model and maintaining well-calibrated uncertainties. This property makes them highly robust under missing input features and enables them to naturally detect outliers and peculiar samples.

Motivation & Objective

• To address the limitations of existing probabilistic deep learning models that cannot compute exact likelihoods during inference.
• To overcome the structural rigidity of traditional SPNs, which hinders scalable training and limits their adoption as deep learning models.
• To enable end-to-end training of SPN-like architectures using standard deep learning tools such as automatic differentiation and SGD.
• To preserve the generative interpretability and uncertainty calibration of SPNs while achieving performance comparable to deep neural networks.
• To enhance model robustness under missing input features and improve outlier detection capabilities.

Proposed method

• The paper proposes random sum-product networks (RAT-SPNs), where SPN structures are randomly initialized and trained using standard deep learning optimization techniques.
• RAT-SPNs employ automatic differentiation and stochastic gradient descent (SGD) for end-to-end training on GPU-accelerated hardware.
• The architecture uses random structures to simplify the learning process, avoiding the need for complex structural constraints typical in traditional SPNs.
• The model maintains the key SPN property of efficient exact inference, including marginalization and conditioning operations.
• The random initialization allows for scalable training while preserving the ability to compute exact data likelihoods.
• The framework integrates generative modeling with discriminative training, enabling both uncertainty estimation and high predictive accuracy.

Experimental results

Research questions

• RQ1 Can SPNs be trained effectively using standard deep learning optimization techniques like SGD and automatic differentiation?
• RQ2 Can random SPN structures maintain exact likelihood computation and efficient inference while enabling scalable training?
• RQ3 How does the performance of RAT-SPNs compare to standard deep neural networks in terms of predictive accuracy?
• RQ4 To what extent do RAT-SPNs maintain well-calibrated uncertainty and robustness under missing input features?
• RQ5 Can RAT-SPNs naturally detect outliers and anomalous samples due to their generative nature?

Key findings

• RAT-SPNs achieve prediction performance comparable to standard deep neural networks on benchmark tasks.
• The models maintain exact computation of data likelihoods, enabling principled probabilistic inference.
• RAT-SPNs demonstrate well-calibrated uncertainty estimates, which enhances reliability in real-world deployment.
• The models remain robust to missing input features, preserving predictive performance under data corruption.
• RAT-SPNs naturally detect outliers and unusual samples due to their generative modeling capability.
• The integration of SPN structures with deep learning training pipelines enables scalable, interpretable, and uncertainty-aware models.