[Paper Review] Non-informative reparameterisations for location-scale mixtures

Kamary, Kaniav, Lee, Jeong Eun+1 more|arXiv (Cornell University)|Jan 6, 2016

Bayesian Methods and Mixture Models32 references3 citations

TL;DR

This paper introduces a novel reparameterization for location-scale finite mixture models by expressing component parameters in terms of the global mean and variance of the mixture, enabling the use of weakly informative uniform priors on compactified parameters. The method ensures proper posteriors with minimal sample sizes and supports efficient MCMC sampling that handles label switching, establishing a reference Bayesian framework for mixtures.

ABSTRACT

While mixtures of Gaussian distributions have been studied for more than a century (Pearson, 1894), the construction of a reference Bayesian analysis of those models still remains unsolved, with a general prohibition of the usage of improper priors (Fr\\"uwirth-Schnatter, 2006) due to the ill-posed nature of such statistical objects. This difficulty is usually bypassed by an empirical Bayes resolution (Richardson and Green, 1997). By creating a new parameterisation cantered on the mean and variance of the mixture distribution itself, we are able to develop here a genuine non-informative prior for Gaussian mixtures with an arbitrary number of components. We demonstrate that the posterior distribution associated with this prior is almost surely proper and provide MCMC implementations that exhibit the expected exchangeability. While we only study here the Gaussian case, extension to other classes of location-scale mixtures is straightforward.

Motivation & Objective

• Address the long-standing challenge of constructing reference Bayesian priors for finite location-scale mixture models, which are typically plagued by improper posteriors under non-informative priors.
• Overcome the identifiability and label switching issues inherent in mixture models by reparameterizing components relative to global mixture parameters.
• Develop a principled, weakly informative prior framework that ensures posterior propriety with minimal sample size, avoiding data-dependent or improper priors.
• Enable robust MCMC inference through compact reparameterization that facilitates uniform and proper priors on transformed parameters.
• Provide a foundation for objective Bayesian analysis of mixtures by deriving a reference prior that is invariant to reparameterization and supports simulation-based inference.

Proposed method

• Reparameterize each component's location and scale parameters as deviations from the global mean and variance of the mixture distribution.
• Transform the component parameters into a compact parameter space using polar-like coordinates relative to the global mean and variance, ensuring boundedness.
• Apply a uniform prior on the compactified parameters (e.g., relative location and scale deviations), which induces a weakly informative prior on the original parameters.
• Derive the joint prior from the Jeffreys prior on the original location-scale parameters, transformed via the new reparameterization to ensure invariance and propriety.
• Implement MCMC algorithms using Metropolis-within-Gibbs sampling, with post-MCMC relabeling via k-means clustering on the parameter space to resolve label switching.
• Validate the approach using the Ultimixt R package, demonstrating convergence and accurate estimation across Gaussian, Poisson, and exponential mixtures.

Experimental results

Research questions

• RQ1 Can a non-informative prior be constructed for finite location-scale mixture models that ensures posterior propriety without relying on data-dependent hyperparameters?
• RQ2 How can reparameterization be used to transform unbounded mixture parameters into a compact space to allow for uniform priors and weakly informative inference?
• RQ3 To what extent does the proposed reparameterization mitigate the label switching problem in MCMC sampling of mixture models?
• RQ4 Does the resulting posterior distribution remain proper under minimal sample sizes, and can it be reliably simulated using standard MCMC methods?
• RQ5 Can this framework be extended to non-Gaussian location-scale families such as Poisson and exponential mixtures while preserving posterior propriety and simulation efficiency?

Key findings

• The reparameterization of component parameters in terms of global mean and variance transforms the parameter space into a compact set, enabling the use of uniform priors on the transformed parameters.
• A weakly informative prior based on the Jeffreys prior on the original location-scale parameters leads to a proper posterior distribution for a minimal sample size, even when the original prior would yield an improper posterior.
• MCMC sampling with the new parameterization successfully handles label switching, as demonstrated by post-MCMC relabeling via k-means clustering producing estimates close to the true values.
• For Gaussian mixtures, the method recovers tissue composition components (e.g., 59% muscle, 33% fat) consistent with prior studies, with the largest component (34%) corresponding to the primary quantity of interest.
• In Poisson mixture examples, the MCMC estimates of component means and weights converge to the true values as sample size increases, with empirical densities from 50,000 simulations closely matching the true parameters.
• The Ultimixt R package successfully implements the method, showing stable convergence and reliable inference across multiple distributions, including compound extensions like the negative binomial.