[Paper Review] Guaranteed Bounds for Posterior Inference in Universal Probabilistic Programming

Raven Beutner, C.-H. Luke Ong+1 more|arXiv (Cornell University)|Apr 6, 2022

Machine Learning and Algorithms4 citations

TL;DR

This paper introduces a sound and complete method for computing guaranteed lower and upper bounds on the posterior distribution of probabilistic programs using interval-based trace semantics and a weight-aware interval type system. The approach ensures the true posterior lies between these bounds, enabling verification of stochastic inference tools like Pyro and Anglican, with a prototype tool (GuBPI) demonstrating practical utility on recursive, higher-order models with continuous distributions.

ABSTRACT

We propose a new method to approximate the posterior distribution of probabilistic programs by means of computing guaranteed bounds. The starting point of our work is an interval-based trace semantics for a recursive, higher-order probabilistic programming language with continuous distributions. Taking the form of (super-/subadditive) measures, these lower/upper bounds are non-stochastic and provably correct: using the semantics, we prove that the actual posterior of a given program is sandwiched between the lower and upper bounds (soundness); moreover the bounds converge to the posterior (completeness). As a practical and sound approximation, we introduce a weight-aware interval type system, which automatically infers interval bounds on not just the return value but also weight of program executions, simultaneously. We have built a tool implementation, called GuBPI, which automatically computes these posterior lower/upper bounds. Our evaluation on examples from the literature shows that the bounds are useful, and can even be used to recognise wrong outputs from stochastic posterior inference procedures.

Motivation & Objective

• To address the lack of correctness guarantees in stochastic posterior inference for universal probabilistic programming languages.
• To provide sound and complete bounds on posterior distributions for recursive, higher-order programs with continuous distributions.
• To enable automated detection of incorrect outputs from existing inference tools like HMC and importance sampling.
• To develop a type system that simultaneously infers bounds on return values and execution weights for improved precision.
• To implement and evaluate a tool (GuBPI) that computes these bounds efficiently and reliably.

Proposed method

• Formalizes a recursive, higher-order probabilistic language with continuous distributions using interval-based trace semantics.
• Defines lower and upper bounds as super- and subadditive measures that sandwich the true posterior, ensuring soundness.
• Introduces a weight-aware interval type system that tracks bounds on both return values and execution weights via constraint-based inference.
• Employs symbolic execution to propagate interval bounds through program traces, enabling static analysis of posterior distributions.
• Uses interval arithmetic to perform safe, conservative computations over probabilistic traces without sampling.
• Implements the GuBPI tool to automatically compute and validate posterior bounds on benchmark models.

Experimental results

Research questions

• RQ1 Can we formally define and compute guaranteed lower and upper bounds on the posterior distribution of a universal probabilistic program?
• RQ2 How can we ensure that the true posterior is always contained within these bounds, regardless of program complexity?
• RQ3 Can the proposed type system simultaneously infer bounds on both return values and execution weights to improve precision?
• RQ4 To what extent can these bounds detect errors in existing stochastic inference tools like HMC or importance sampling?
• RQ5 How efficiently can the method scale to recursive and higher-order models with continuous distributions?

Key findings

• The proposed interval-based trace semantics ensures soundness: the true posterior is always bounded between the computed lower and upper bounds.
• Completeness is achieved: as refinement increases, the bounds converge to the true posterior distribution.
• The weight-aware interval type system enables precise, automated inference of bounds on both return values and execution weights.
• GuBPI successfully detects incorrect outputs from Pyro’s HMC and Anglican’s importance sampling on the pedestrian example, where results disagreed significantly.
• Simulation-based calibration confirmed that HMC missed posterior modes, while GuBPI’s bounds remained correct and verifiable.
• GuBPI outperforms simulation-based calibration in runtime for models like the pedestrian example, while providing formal guarantees.