[Paper Review] Declarative Probabilistic Programming with Datalog

Vikash K. Mansinghka, Daniel Selsam+1 more|arXiv (Cornell University)|Apr 1, 2014

Bayesian Modeling and Causal InferenceComputer Science44 references180 citations

TL;DR

This paper introduces Venture, a Turing-complete, higher-order probabilistic programming language that supports customizable inference strategies through a novel stochastic procedure interface (SPI), probabilistic execution traces (PETs), scaffolds for incremental inference, and stochastic regeneration algorithms. The key contribution is a scalable, composable inference framework that enables efficient Metropolis-Hastings, Gibbs sampling, and hybrid particle-Markov chain Monte Carlo methods with linear-time complexity in practice, overcoming quadratic scaling issues in prior approaches.

ABSTRACT

Probabilistic programming languages are used for developing statistical models, and they typically consist of two components: a specification of a stochastic process (the prior), and a specification of observations that restrict the probability space to a conditional subspace (the posterior). Use cases of such formalisms include the development of algorithms in machine learning and artificial intelligence. We propose and investigate an extension of Datalog for specifying statistical models, and establish a declarative probabilistic-programming paradigm over databases. Our proposed extension provides convenient mechanisms to include common numerical probability functions; in particular, conclusions of rules may contain values drawn from such functions. The semantics of a program is a probability distribution over the possible outcomes of the input database with respect to the program. Observations are naturally incorporated by means of integrity constraints over the extensional and intensional relations. The resulting semantics is robust under different chases and invariant to rewritings that preserve logical equivalence.

Motivation & Objective

• To design a general-purpose probabilistic programming system that supports both expressive modeling and customizable inference.
• To address the inefficiency of existing inference methods that scale quadratically with model complexity.
• To enable integration of external models and likelihood-free simulators within a unified inference framework.
• To provide a composable, extensible architecture for building hybrid inference strategies combining MCMC and variational inference.
• To support both automatic inference and expert-controlled inference programming for diverse application workloads.

Proposed method

• Introduces the Stochastic Procedure Interface (SPI) to encapsulate primitive random variables with custom control flow, higher-order procedures, and latent simulation requests.
• Defines Probabilistic Execution Traces (PETs) as data structures capturing conditional, existential, and exchangeable dependencies in program execution.
• Proposes scaffolds—partitions of PETs—that factor global inference into coherent, local sub-problems for incremental processing.
• Develops stochastic regeneration algorithms that modify PET fragments within scaffolds without revisiting conditionally independent variables, enabling linear-time updates.
• Uses regeneration to build invariant transition operators for Metropolis-Hastings and hybrid particle-MCMC methods.
• Supports inference programming via composable kernels and state-dependent proposal selection, enabling general-purpose inference strategies.

Experimental results

Research questions

• RQ1 Can a probabilistic programming system support both Turing-complete modeling and composable, efficient inference strategies at scale?
• RQ2 How can inference algorithms be made robust to changes in program structure without incurring quadratic computational costs?
• RQ3 Can stochastic regeneration enable efficient, context-independent updates to execution traces during MCMC sampling?
• RQ4 How can likelihood-free simulators and external models be natively integrated into a probabilistic inference pipeline?
• RQ5 Can hybrid inference strategies combining MCMC and variational inference be implemented with composability and performance guarantees?

Key findings

• Stochastic regeneration enables inference updates that scale linearly with model size, avoiding the quadratic blowup common in prior approaches.
• The scaffold-based decomposition of execution traces allows for efficient, incremental inference by isolating local dependencies.
• The SPI enables seamless integration of external models and likelihood-free simulators with custom proposals and latent variable management.
• Hybrid inference strategies such as particle-Markov chain Monte Carlo and mean-field variational inference can be implemented using the same composable kernel framework.
• The framework supports both automatic inference and expert-programmed inference, enabling a spectrum from black-box to highly tuned inference strategies.
• Empirical results demonstrate that stochastic regeneration reduces inference time significantly in complex models with dynamic structure, particularly in models involving partially exchangeable sequences and nested stochastic processes.