[Paper Review] Priors for Random Count Matrices Derived from a Family of Negative Binomial Processes

Mingyuan Zhou, Oscar Hernán Madrid Padilla+1 more|arXiv (Cornell University)|Apr 12, 2014

Bayesian Methods and Mixture Models31 references4 citations

TL;DR

This paper introduces a family of nonparametric Bayesian priors for random count matrices using gamma-Poisson, gamma-negative binomial, and beta-negative binomial processes. These models enable closed-form Gibbs sampling, support unbounded feature spaces, and outperform traditional methods in text classification by naturally handling unseen features without vocabulary predefinition or parameter tuning.

ABSTRACT

We define a family of probability distributions for random count matrices with a potentially unbounded number of rows and columns. The three distributions we consider are derived from the gamma-Poisson, gamma-negative binomial, and beta-negative binomial processes. Because the models lead to closed-form Gibbs sampling update equations, they are natural candidates for nonparametric Bayesian priors over count matrices. A key aspect of our analysis is the recognition that, although the random count matrices within the family are defined by a row-wise construction, their columns can be shown to be i.i.d. This fact is used to derive explicit formulas for drawing all the columns at once. Moreover, by analyzing these matrices' combinatorial structure, we describe how to sequentially construct a column-i.i.d. random count matrix one row at a time, and derive the predictive distribution of a new row count vector with previously unseen features. We describe the similarities and differences between the three priors, and argue that the greater flexibility of the gamma- and beta- negative binomial processes, especially their ability to model over-dispersed, heavy-tailed count data, makes these well suited to a wide variety of real-world applications. As an example of our framework, we construct a naive-Bayes text classifier to categorize a count vector to one of several existing random count matrices of different categories. The classifier supports an unbounded number of features, and unlike most existing methods, it does not require a predefined finite vocabulary to be shared by all the categories, and needs neither feature selection nor parameter tuning. Both the gamma- and beta- negative binomial processes are shown to significantly outperform the gamma-Poisson process for document categorization, with comparable performance to other state-of-the-art supervised text classification algorithms.

Motivation & Objective

• Address the lack of flexible nonparametric Bayesian priors for random count matrices with potentially unbounded dimensions.
• Develop a framework that supports predictive distributions for new rows with previously unseen features in count matrices.
• Enable scalable, parallel inference for count matrices across multiple categories without requiring a shared finite vocabulary.
• Construct a naive-Bayes classifier that operates directly on raw counts and handles over-dispersion and heavy-tailed data.
• Demonstrate superior performance in document categorization compared to standard multinomial naive Bayes and other state-of-the-art models.

Proposed method

• Derive three probability distributions for random count matrices from gamma-Poisson, gamma-negative binomial, and beta-negative binomial processes.
• Show that columns of the resulting matrices are i.i.d., enabling joint sampling or sequential row-by-row construction.
• Use combinatorial analysis to derive the predictive distribution of a new row vector with unseen features.
• Implement closed-form Gibbs sampling updates for all three models, enabling efficient MCMC inference.
• Construct a nonparametric Bayesian naive-Bayes classifier that directly uses raw word counts and avoids vocabulary predefinition.
• Extend the framework to online learning and support unbounded feature spaces through nonparametric priors.

Experimental results

Research questions

• RQ1 How can we define a nonparametric Bayesian prior for random count matrices with potentially infinite rows and columns?
• RQ2 What is the predictive distribution of a new row vector containing previously unseen features in a count matrix?
• RQ3 How do the gamma-negative binomial and beta-negative binomial processes compare to the gamma-Poisson process in modeling over-dispersed, heavy-tailed count data?
• RQ4 Can a nonparametric Bayesian naive-Bayes classifier be constructed that avoids vocabulary predefinition and parameter tuning?
• RQ5 What performance gains are achievable in document categorization using these new priors compared to standard multinomial models with Laplace smoothing?

Key findings

• The gamma-negative binomial process and beta-negative binomial process significantly outperform the gamma-Poisson process in document categorization tasks.
• Both GNBP and BNBP achieve performance comparable to state-of-the-art discriminatively trained text classification algorithms.
• The proposed classifiers require no feature selection or parameter tuning, and naturally handle previously unseen words in test documents.
• Even with a single MCMC sample ($S=1$), the predictive likelihoods yield stable and accurate categorization results with low Monte Carlo variability.
• The boxplot analysis shows that increasing $S$ reduces variability but provides no substantial improvement in mean accuracy, indicating that small $S$ suffices in practice.
• The models’ ability to marginalize over latent processes and derive explicit PMFs for i.i.d. columns enables robust and scalable inference.