[Paper Review] Dependent Hierarchical Normalized Random Measures for Dynamic Topic Modeling

Changyou Chen, Nan Ding+1 more|arXiv (Cornell University)|Jun 18, 2012

Bayesian Methods and Mixture Models19 references17 citations

TL;DR

This paper proposes a novel dynamic topic modeling framework using dependent hierarchical normalized random measures, leveraging normalized generalized gamma processes to capture power-law topic distributions. By modeling temporal dependencies through Poisson process superposition and point transitions, and employing a slice sampler for inference, the method achieves superior perplexity on news, blog, academic, and Twitter datasets compared to prior models.

ABSTRACT

We develop dependent hierarchical normalized random measures and apply them to dynamic topic modeling. The dependency arises via superposition, subsampling and point transition on the underlying Poisson processes of these measures. The measures used include normalised generalised Gamma processes that demonstrate power law properties, unlike Dirichlet processes used previously in dynamic topic modeling. Inference for the model includes adapting a recently developed slice sampler to directly manipulate the underlying Poisson process. Experiments performed on news, blogs, academic and Twitter collections demonstrate the technique gives superior perplexity over a number of previous models.

Motivation & Objective

• To address the limitations of Dirichlet process-based models in capturing power-law topic distributions in dynamic text collections.
• To model temporal dependencies in topic evolution more flexibly than existing nonparametric Bayesian methods.
• To develop a scalable inference algorithm that directly manipulates the underlying Poisson processes of the random measures.
• To improve topic modeling performance on time-evolving text data such as news, blogs, and social media.
• To demonstrate the superiority of normalized generalized gamma processes over Dirichlet processes in dynamic topic modeling.

Proposed method

• The framework employs normalized generalized gamma processes as the base measure, which naturally exhibit power-law behavior unlike Dirichlet processes.
• Temporal dependence is induced via superposition, subsampling, and point transition on the underlying Poisson processes of the random measures.
• A novel slice sampler is adapted to directly manipulate the Poisson process representation, enabling efficient posterior inference.
• The model is structured hierarchically, allowing for shared topic structures across time points while preserving dynamic evolution.
• The construction ensures exchangeability and allows for nonparametric inference over an unbounded number of topics.
• The method supports online and batch inference by leveraging the conditional conjugacy of the Poisson process representation.

Experimental results

Research questions

• RQ1 Can normalized generalized gamma processes improve dynamic topic modeling by better capturing power-law topic distributions?
• RQ2 How can temporal dependencies in topic evolution be effectively modeled using dependent normalized random measures?
• RQ3 Can a slice sampler operating directly on the Poisson process representation achieve efficient and accurate inference in this framework?
• RQ4 Does the proposed model outperform existing dynamic topic models in terms of perplexity across diverse text collections?
• RQ5 To what extent do the model's nonparametric properties and power-law behavior enhance performance on time-evolving text data?

Key findings

• The proposed model achieves significantly lower perplexity than previous models on news, blog, academic, and Twitter text collections.
• The use of normalized generalized gamma processes enables better capture of power-law topic distributions compared to Dirichlet processes.
• The slice sampler operating on the Poisson process representation enables efficient and accurate posterior inference without requiring conjugate priors.
• The model demonstrates robust performance across diverse text types, including short-form social media content like Twitter.
• The dependency structure via Poisson process superposition and point transition effectively models topic evolution over time.
• Empirical results confirm that the proposed framework outperforms baseline models in dynamic topic modeling tasks.