[论文解读] Priors for Random Count Matrices Derived from a Family of Negative Binomial Processes

Mingyuan Zhou, Oscar Hernán Madrid Padilla+1 more|arXiv (Cornell University)|Apr 12, 2014

Bayesian Methods and Mixture Models参考文献 31被引用 4

一句话总结

本文提出了一类基于伽马-泊松、伽马-负二项分布和贝塔-负二项分布过程的非参数贝叶斯先验，用于随机计数矩阵。这些模型支持闭式吉布斯采样，可处理无界特征空间，并在文本分类任务中优于传统方法，能自然地处理未见特征而无需预定义词汇表或超参数调优。

ABSTRACT

We define a family of probability distributions for random count matrices with a potentially unbounded number of rows and columns. The three distributions we consider are derived from the gamma-Poisson, gamma-negative binomial, and beta-negative binomial processes. Because the models lead to closed-form Gibbs sampling update equations, they are natural candidates for nonparametric Bayesian priors over count matrices. A key aspect of our analysis is the recognition that, although the random count matrices within the family are defined by a row-wise construction, their columns can be shown to be i.i.d. This fact is used to derive explicit formulas for drawing all the columns at once. Moreover, by analyzing these matrices' combinatorial structure, we describe how to sequentially construct a column-i.i.d. random count matrix one row at a time, and derive the predictive distribution of a new row count vector with previously unseen features. We describe the similarities and differences between the three priors, and argue that the greater flexibility of the gamma- and beta- negative binomial processes, especially their ability to model over-dispersed, heavy-tailed count data, makes these well suited to a wide variety of real-world applications. As an example of our framework, we construct a naive-Bayes text classifier to categorize a count vector to one of several existing random count matrices of different categories. The classifier supports an unbounded number of features, and unlike most existing methods, it does not require a predefined finite vocabulary to be shared by all the categories, and needs neither feature selection nor parameter tuning. Both the gamma- and beta- negative binomial processes are shown to significantly outperform the gamma-Poisson process for document categorization, with comparable performance to other state-of-the-art supervised text classification algorithms.

研究动机与目标

• 解决随机计数矩阵在潜在无界维度下缺乏灵活的非参数贝叶斯先验的问题。
• 构建一个支持计数矩阵中新增行向量（含未见特征）预测分布的框架。
• 实现无需共享有限词汇表的多类别计数矩阵的可扩展、并行推断。
• 构建一个直接基于原始计数的朴素贝叶斯分类器，能够处理过度离散和重尾数据。
• 在文档分类任务中表现优于标准多项式朴素贝叶斯及其他最先进模型。

提出的方法

• 从伽马-泊松、伽马-负二项分布和贝塔-负二项分布过程推导出三种用于随机计数矩阵的概率分布。
• 证明所得矩阵的列是独立同分布的，从而支持联合采样或逐行构建。
• 通过组合分析推导出包含未见特征的新行向量的预测分布。
• 为所有三种模型实现闭式吉布斯采样更新，支持高效的马尔可夫链蒙特卡洛推断。
• 构建一个直接使用原始词频的非参数贝叶斯朴素贝叶斯分类器，避免词汇表预定义。
• 将框架扩展至在线学习，并通过非参数先验支持无界特征空间。

实验结果

研究问题

• RQ1 如何为具有潜在无限行和列的随机计数矩阵定义非参数贝叶斯先验？
• RQ2 在计数矩阵中，包含未见特征的新行向量的预测分布是什么？
• RQ3 伽马-负二项分布过程与贝塔-负二项分布过程相较于伽马-泊松过程，在建模过度离散、重尾计数数据方面表现如何？
• RQ4 能否构建一个避免词汇表预定义和超参数调优的非参数贝叶斯朴素贝叶斯分类器？
• RQ5 与采用拉普拉斯平滑的标准多项式模型相比，使用这些新先验在文档分类中可实现多大的性能提升？

主要发现

• 伽马-负二项分布过程（GNBP）和贝塔-负二项分布过程（BNBP）在文档分类任务中显著优于伽马-泊松过程。
• GNBP与BNBP的性能可与最先进的判别式训练文本分类算法相媲美。
• 所提出的分类器无需特征选择或超参数调优，且能自然处理测试文档中的未见词汇。
• 即使仅使用单次MCMC样本（$S=1$），预测似然值也能提供稳定且准确的分类结果，蒙特卡洛变异性极低。
• 箱线图分析表明，增加$S$可降低变异性，但对平均准确率无显著提升，表明实践中小$S$已足够。
• 模型通过边际化潜在过程并推导出独立同分布列的显式概率质量函数，实现了稳健且可扩展的推断。