[Paper Review] Variable Selection for Latent Dirichlet Allocation

Dongwoo Kim, Yeonseung Chung+1 more|arXiv (Cornell University)|May 4, 2012

Natural Language Processing Techniques25 references3 citations

TL;DR

This paper proposes vsLDA, a variable selection method for Latent Dirichlet Allocation that automatically identifies and excludes non-informative words from the vocabulary during topic inference. By modeling word informativity via Bernoulli-distributed selection variables, vsLDA improves topic discriminability and robustness, outperforming symmetric and asymmetric LDA variants in document classification and MCMC consistency, with Jaccard similarity of 0.83–0.96 for non-informative word sets across runs.

ABSTRACT

In latent Dirichlet allocation (LDA), topics are multinomial distributions over the entire vocabulary. However, the vocabulary usually contains many words that are not relevant in forming the topics. We adopt a variable selection method widely used in statistical modeling as a dimension reduction tool and combine it with LDA. In this variable selection model for LDA (vsLDA), topics are multinomial distributions over a subset of the vocabulary, and by excluding words that are not informative for finding the latent topic structure of the corpus, vsLDA finds topics that are more robust and discriminative. We compare three models, vsLDA, LDA with symmetric priors, and LDA with asymmetric priors, on heldout likelihood, MCMC chain consistency, and document classification. The performance of vsLDA is better than symmetric LDA for likelihood and classification, better than asymmetric LDA for consistency and classification, and about the same in the other comparisons.

Motivation & Objective

• To address the ad-hoc and arbitrary nature of traditional vocabulary preprocessing (e.g., stopword removal) in LDA.
• To develop a principled, model-based variable selection method that jointly selects informative words and infers topics.
• To improve topic model robustness and predictive power by excluding words that hinder topic discovery.
• To provide a systematic alternative to fixed vocabulary selection, enabling data-driven word subset identification.
• To evaluate vsLDA against symmetric and asymmetric LDA in terms of likelihood, consistency, and classification performance.

Proposed method

• Introduces a generative process where each word type is assigned a binary indicator $ s_j \sim \text{Bernoulli}(\lambda) $ to determine if it is included in the topic vocabulary.
• Replaces the standard Dirichlet prior $ \text{Dir}(\beta\mathbf{1}) $ with $ \text{Dir}(\beta\mathbf{s}) $, so topics are defined only over the selected subset of words.
• Uses Markov Chain Monte Carlo (MCMC) with Gibbs sampling for posterior inference over latent variables, including topic assignments and word selection indicators.
• Applies Monte Carlo integration to approximate the posterior distribution over the selected word set and topic distributions.
• Employs a Hungarian algorithm-based best-matching method to compute symmetric KL divergence between topic pairs across multiple MCMC chains for consistency evaluation.
• Uses Jaccard coefficient to measure overlap of non-informative (NI) word sets across multiple MCMC runs as a proxy for stability.

Experimental results

Research questions

• RQ1 Can a model-based variable selection approach improve topic model performance by excluding non-informative words during topic inference?
• RQ2 How does vsLDA compare to symmetric and asymmetric LDA in terms of held-out likelihood and predictive performance?
• RQ3 Does vsLDA produce more consistent topic estimates across multiple MCMC runs than asymmetric LDA?
• RQ4 To what extent does the exclusion of non-informative words enhance document classification accuracy?
• RQ5 How stable is the set of non-informative words identified by vsLDA across multiple MCMC chains?

Key findings

• vsLDA achieves better held-out likelihood than symmetric LDA and better document classification accuracy than both symmetric and asymmetric LDA.
• vsLDA demonstrates higher MCMC chain consistency than asymmetric LDA, with average symmetric KL divergence between best-matching topic pairs being lower than for asymmetric LDA.
• The average Jaccard similarity of non-informative word sets across multiple MCMC runs is 0.83 for 20NG, 0.96 for NIPS, and 0.93 for SigGraph, indicating high stability.
• vsLDA reduces each document into more discriminative subdimensions, leading to superior classification performance compared to both symmetric and asymmetric LDA.
• vsLDA performs comparably to symmetric LDA in terms of symmetric KL divergence and consistency, despite using asymmetric priors.
• The model effectively reduces the influence of non-informative words that are typically removed via arbitrary preprocessing, such as stopword lists.