[Paper Review] Church: a language for generative models

Noah D. Goodman, Vikash K. Mansinghka+3 more|arXiv (Cornell University)|Jun 13, 2012

Bayesian Modeling and Causal InferenceComputer Science21 references507 citations

TL;DR

Church is a probabilistic programming language based on Lisp that enables the specification of complex generative models using a stochastic lambda calculus. It introduces the stochastic memoizer to simplify non-parametric models and supports exact and approximate inference via Monte Carlo methods, enabling flexible, scalable Bayesian reasoning across diverse AI applications including clustering, planning, and hierarchical modeling.

ABSTRACT

We introduce Church, a universal language for describing stochastic generative processes. Church is based on the Lisp model of lambda calculus, containing a pure Lisp as its deterministic subset. The semantics of Church is defined in terms of evaluation histories and conditional distributions on such histories. Church also includes a novel language construct, the stochastic memoizer, which enables simple description of many complex non-parametric models. We illustrate language features through several examples, including: a generalized Bayes net in which parameters cluster over trials, infinite PCFGs, planning by inference, and various non-parametric clustering models. Finally, we show how to implement query on any Church program, exactly and approximately, using Monte Carlo techniques.

Motivation & Objective

• To design a universal language for describing stochastic generative processes in a principled, composable way.
• To integrate probabilistic reasoning with functional programming through a Lisp-based syntax and semantics.
• To enable concise modeling of non-parametric Bayesian models, such as infinite mixture models and clustering.
• To support both exact and approximate inference via Monte Carlo techniques for practical deployment.
• To provide a formal semantics grounded in evaluation histories and conditional distributions for sound model interpretation.

Proposed method

• Church extends Lisp with stochastic primitives, allowing random choices to be embedded in functional expressions.
• It defines semantics via evaluation histories and conditional distributions over these histories to model probabilistic computation.
• The stochastic memoizer is introduced as a core language construct to cache and reuse random values, enabling efficient modeling of non-parametric distributions.
• The language supports higher-order functions and closures, allowing recursive and parametric modeling of complex stochastic processes.
• Inference is implemented using Markov Chain Monte Carlo (MCMC) and other Monte Carlo techniques to answer queries on Church programs.
• The system supports both exact inference (via enumeration) and approximate inference (via sampling), with extensibility to new inference algorithms.

Experimental results

Research questions

• RQ1 How can a general-purpose programming language be extended with first-class stochastic primitives to model arbitrary generative processes?
• RQ2 What formal semantics supports sound and composable probabilistic reasoning in a functional language?
• RQ3 How can non-parametric models like infinite mixture models be expressed concisely and efficiently?
• RQ4 Can a single language and inference framework unify diverse probabilistic modeling tasks, including planning and hierarchical modeling?
• RQ5 What mechanisms enable scalable and extensible inference in complex, recursive probabilistic programs?

Key findings

• Church enables the specification of complex non-parametric models, such as infinite PCFGs and clustering models, with minimal syntactic overhead.
• The stochastic memoizer allows for natural and efficient encoding of models with shared random variables and infinite-dimensional distributions.
• The language supports both exact and approximate inference, with Monte Carlo methods providing scalable solutions for large or complex models.
• Church's semantics are formally grounded in evaluation histories and conditional distributions, ensuring sound interpretation of probabilistic programs.
• The framework successfully models hierarchical Bayesian networks with clustering over parameters across trials, demonstrating flexibility in modeling structured uncertainty.
• The integration of functional programming and probabilistic reasoning enables expressive, modular, and reusable probabilistic models across diverse AI applications.