[论文解读] A survey on Bayesian inference for Gaussian mixture model

Jun Lu|arXiv (Cornell University)|Aug 20, 2021

Bayesian Methods and Mixture Models参考文献 60被引用 7

一句话总结

本综述为有限和无限高斯混合模型的贝叶斯推断提供了全面且自包含的介绍，涵盖共轭先验、狄利克雷过程和吉布斯采样等基础概念。它强调理论严谨性，包含详细的推导过程，并通过实用的推断技术和度量指标突出现代应用。

ABSTRACT

Clustering has become a core technology in machine learning, largely due to its application in the field of unsupervised learning, clustering, classification, and density estimation. A frequentist approach exists to hand clustering based on mixture model which is known as the EM algorithm where the parameters of the mixture model are usually estimated into a maximum likelihood estimation framework. Bayesian approach for finite and infinite Gaussian mixture model generates point estimates for all variables as well as associated uncertainty in the form of the whole estimates' posterior distribution. The sole aim of this survey is to give a self-contained introduction to concepts and mathematical tools in Bayesian inference for finite and infinite Gaussian mixture model in order to seamlessly introduce their applications in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results concerning this field and given the paucity of scope to present this discussion, e.g., the separated analysis of the generation of Dirichlet samples by stick-breaking and Polya's Urn approaches. We refer the reader to literature in the field of the Dirichlet process mixture model for a much detailed introduction to the related fields. Some excellent examples include (Frigyik et al., 2010; Murphy, 2012; Gelman et al., 2014; Hoff, 2009). This survey is primarily a summary of purpose, significance of important background and techniques for Gaussian mixture model, e.g., Dirichlet prior, Chinese restaurant process, and most importantly the origin and complexity of the methods which shed light on their modern applications. The mathematical prerequisite is a first course in probability. Other than this modest background, the development is self-contained, with rigorous proofs provided throughout.

研究动机与目标

• 为有限和无限高斯混合模型的贝叶斯推断提供一个自包含且数学严谨的介绍。
• 阐明共轭先验、中国餐馆过程和狄利克雷过程等关键工具在建模不确定性和聚类中的作用。
• 完整推导并展示折叠吉布斯采样和自适应拒绝采样等基础技术。
• 通过推断复杂度分析和优化策略（如Cholesky分解和剪枝方法），提供对现代应用的深入见解。
• 在仅需基本概率知识的前提下，引导研究人员理解后验集中性和渐近行为等理论性质。

提出的方法

• 使用共轭先验——特别是正态-逆维施特分布（NIW）——对多变量高斯分量的均值和协方差进行联合估计。
• 应用折叠吉布斯采样以边际化聚类参数，通过积分掉均值和协方差来简化后验推断。
• 采用自适应拒绝采样（ARS）从对数凹后验密度中高效采样，尤其适用于高维情形。
• 引入中国餐馆过程（CRP）和狄利克雷过程（DP）作为非参数先验，以建模未知且可能无限的聚类数量。
• 利用Cholesky分解提高矩阵运算的计算效率，包括行列式计算和秩一更新。
• 应用剪枝技术（如约束采样（cSampling）和基于损失的采样（lSampling））以提升无限混合模型中的可扩展性。

实验结果

研究问题

• RQ1 如何系统性地应用共轭先验，以推导出高斯混合模型的闭式后验？
• RQ2 在有限模型中，对称狄利克雷先验下混合权重的后验分布具有哪些理论性质？
• RQ3 中国餐馆过程如何实现未知分量数的非参数贝叶斯聚类？
• RQ4 在有限和无限高斯混合模型中，使用折叠吉布斯采样时存在哪些计算与统计权衡？
• RQ5 优化技术（如Cholesky分解和剪枝）如何提升大规模场景下后验推断的效率？

主要发现

• 正态-逆维施特（NIW）先验可为多变量高斯分量的均值和协方差提供闭式后验，从而实现精确的贝叶斯推断。
• 折叠吉布斯采样通过积分掉聚类参数降低计算成本，其复杂度在基于Cholesky更新时为O(n³)。
• 中国餐馆过程提供了狄利克雷过程的棒折构造，支持未知分量数的非参数聚类。
• 在共轭先验下，新数据的后预测分布具有解析可处理性，有助于模型评估与预测。
• 函数$ \Gamma(Kx)/[\Gamma(x)]^K $是严格对数凹的，这一结果为狄利克雷过程建模中的理论保证提供了基础。
• 剪枝方法（如cSampling和lSampling）通过在推断过程中消除冗余分量，显著降低了无限混合模型中的计算开销。