[Paper Review] Dynamic mixtures of finite mixtures and telescoping sampling

Sylvia Frühwirth‐Schnatter, Gertraud Malsiner‐Walli+1 more|arXiv (Cornell University)|May 20, 2020

Bayesian Methods and Mixture Models36 references4 citations

TL;DR

This paper introduces dynamic mixtures of finite mixtures (MFMs) with a novel telescoping sampler that enables efficient Bayesian inference for models with arbitrary component distributions. By allowing the Dirichlet concentration parameter to depend on the number of components and proposing a flexible prior on K, the method provides computationally feasible inference while linking MFMs to Bayesian nonparametric mixtures.

ABSTRACT

Within a Bayesian framework, a comprehensive investigation of the model class of mixtures of finite mixtures (MFMs) where a prior on the number of components is specified is performed. This model class has applications in model-based clustering as well as for semi-parametric density approximation, but requires suitable prior specifications and inference methods to exploit its full potential. We contribute to the Bayesian analysis of MFMs by (1) considering static and dynamic MFMs where the Dirichlet parameter of the component weights either is fixed or depends on the number of components, (2) proposing a flexible prior distribution class for the number of components $K$, (3) characterizing the implicit prior on the number of clusters $K_+$ as well as partitions by deriving computationally feasible formulas, (4) linking MFMs to Bayesian non-parametric mixtures, and (5) finally proposing a novel sampling scheme for MFMs called the telescoping sampler which allows Bayesian inference for mixtures with arbitrary component distributions. The telescoping sampler explicitly samples the number of components, but otherwise requires only the usual MCMC steps for estimating a finite mixture model. The ease of its application using different component distributions is demonstrated on real data sets.

Motivation & Objective

• To develop a flexible Bayesian framework for mixtures of finite mixtures (MFMs) that supports both model-based clustering and semi-parametric density estimation.
• To address the challenge of prior specification for the number of components K in MFMs, particularly in terms of inducing appropriate clustering behavior.
• To bridge MFMs with Bayesian nonparametric mixtures by characterizing the implicit prior on the number of clusters K+ and partition structures.
• To design a scalable and general-purpose MCMC sampling scheme—called the telescoping sampler—that explicitly samples K while retaining standard MCMC steps for component estimation.

Proposed method

• Proposes a dynamic MFM model where the Dirichlet concentration parameter for component weights is allowed to depend on the number of components K.
• Introduces a flexible class of priors for the number of components K, enabling control over the prior belief in cluster count.
• Derives computationally feasible formulas for the implicit prior on the number of clusters K+ and on partition structures induced by the MFM prior.
• Establishes theoretical connections between MFMs and Bayesian nonparametric mixtures, particularly in the limit as K increases.
• Develops the telescoping sampler, a novel MCMC algorithm that explicitly samples K and integrates seamlessly with standard finite mixture MCMC updates.
• Applies the telescoping sampler to models with arbitrary component distributions, demonstrating its generality and scalability on real data.

Experimental results

Research questions

• RQ1 How can the Dirichlet concentration parameter in finite mixtures be made adaptive to the number of components to improve clustering performance?
• RQ2 What are the implicit prior distributions on the number of clusters K+ and on partition structures induced by a prior on K in MFMs?
• RQ3 How can MFMs be systematically linked to Bayesian nonparametric mixture models?
• RQ4 Can a general-purpose MCMC sampler be designed for MFMs that supports arbitrary component distributions and efficient inference on K?
• RQ5 What is the empirical performance of the proposed telescoping sampler compared to standard MCMC methods in real-world clustering and density estimation tasks?

Key findings

• The proposed dynamic MFM model allows the Dirichlet concentration parameter to vary with K, leading to more adaptive and realistic clustering behavior.
• The derived formulas for the implicit prior on K+ and partitions enable precise control and interpretation of clustering prior beliefs in MFMs.
• The connection between MFMs and Bayesian nonparametric mixtures is formally established, showing that MFMs can serve as a finite approximation with well-characterized asymptotic properties.
• The telescoping sampler enables efficient Bayesian inference for MFMs with arbitrary component distributions by explicitly sampling K while using standard MCMC steps for other parameters.
• Empirical results on real data demonstrate the practical utility and scalability of the telescoping sampler across diverse component distributions and clustering tasks.