[Paper Review] A Bayesian View of the Poisson-Dirichlet Process

Wray Buntine, Marcus Hütter|arXiv (Cornell University)|Jul 2, 2010

Bayesian Methods and Mixture Models27 references48 citations

TL;DR

This paper presents a Bayesian interpretation of the Poisson-Dirichlet process by deriving a recursive characterization of the distribution of the number of distinct species (M) in a sample of size N. It establishes that the generalized Stirling numbers S(N,M; -1,-a,0) exactly match the normalized probability mass function p(M|N), providing a combinatorial and analytical foundation for the process through recursion and boundary conditions.

ABSTRACT

The two parameter Poisson-Dirichlet Process (PDP), a generalisation of the Dirichlet Process, is increasingly being used for probabilistic modelling in discrete areas such as language technology, bioinformatics, and image analysis. There is a rich literature about the PDP and its derivative distributions such as the Chinese Restaurant Process (CRP). This article reviews some of the basic theory and then the major results needed for Bayesian modelling of discrete problems including details of priors, posteriors and computation. The PDP allows one to build distributions over countable partitions. The PDP has two other remarkable properties: first it is partially conjugate to itself, which allows one to build hierarchies of PDPs, and second using a marginalised relative the CRP, one gets fragmentation and clustering properties that lets one layer partitions to build trees. This article presents the basic theory for understanding the notion of partitions and distributions over them, the PDP and the CRP, and the important properties of conjugacy, fragmentation and clustering, as well as some key related properties such as consistency and convergence. This article also presents a Bayesian interpretation of the Poisson-Dirichlet process based on an improper and infinite dimensional Dirichlet distribution. This means we can understand the process as just another Dirichlet and thus all its sampling properties emerge naturally. The theory of PDPs is usually presented for continuous distributions (more generally referred to as non-atomic distributions), however, when applied to discrete distributions its remarkable conjugacy property emerges. This context and basic results are also presented, as well as techniques for computing the second order Stirling numbers that occur in the posteriors for discrete distributions.

Motivation & Objective

• To provide a Bayesian interpretation of the Poisson-Dirichlet process through the distribution of the number of distinct species in a sample.
• To derive a recursive formula for p(M|N) based on predictive sampling dynamics.
• To establish equivalence between the species count distribution and generalized Stirling numbers S(N,M; -1,-a,0).
• To validate the boundary conditions and asymptotic behavior of the distribution using explicit expressions.

Proposed method

• Derives a recursion for p(M_{N+1} = m | M_N) using the predictive distribution of the Dirichlet process.
• Uses the explicit form p(M_N = m) = S_{m,a}^N (b|a)^m / (b)_N from Lemma LABEL:lem-exp.
• Applies the recursion to derive a recurrence: S_{m,a}^{N+1} = S_{m-1,a}^N + (N - m a) S_{m,a}^N.
• Identifies the generalized Stirling numbers S(n,k; @, β,r) with parameters (-1,-a,0) as matching the species count distribution.
• Verifies boundary conditions S_{m,a}^N = 0 for m > N and S_{0,a}^N = δ_{N,0} via definition and combinatorial interpretation.
• Demonstrates continuity in the limit a → 0 by relating the expression to partial derivatives and interpolation.

Experimental results

Research questions

• RQ1 How can the distribution of the number of distinct species in a sample be characterized using Bayesian nonparametric methods?
• RQ2 What recursive structure underlies the transition probabilities for the number of species as sample size increases?
• RQ3 How do generalized Stirling numbers S(N,M; -1,-a,0) relate to the normalized probability mass function of the species count?
• RQ4 What is the role of the parameters a and b in shaping the species distribution and its recursion?
• RQ5 How does the limit as a → 0 recover the partial derivative form of the species count distribution?

Key findings

• The recursion for p(M_{N+1} = m) is derived from the predictive sampling distribution and matches the recurrence S_{m,a}^{N+1} = S_{m-1,a}^N + (N - m a) S_{m,a}^N.
• The generalized Stirling numbers S(N,M; -1,-a,0) are proven to equal the normalized probability p(M_N = m) under the Poisson-Dirichlet process.
• Boundary conditions S_{m,a}^N = 0 for m > N and S_{0,a}^N = δ_{N,0} are confirmed via the explicit formula and process interpretation.
• The case a = 0 is shown to correspond to the M-th partial derivative through interpolation, linking discrete and continuous forms.
• The equivalence between the species count distribution and the generalized Stirling number expression is rigorously established via parameter substitution and recursive matching.