[Paper Review] MetaLDA: a Topic Model that Efficiently Incorporates Meta information

He Zhao, Lan Du+2 more|arXiv (Cornell University)|Sep 19, 2017

Topic Modeling3 citations

TL;DR

MetaLDA is a topic model that efficiently incorporates document and word meta information—such as labels, categories, and word embeddings—via fully local conjugacy enabled by data augmentation, allowing fast Gibbs sampling. It achieves superior topic quality and lower perplexity, especially on sparse texts, while running significantly faster than existing meta-information-augmented models.

ABSTRACT

Besides the text content, documents and their associated words usually come with rich sets of meta informa- tion, such as categories of documents and semantic/syntactic features of words, like those encoded in word embeddings. Incorporating such meta information directly into the generative process of topic models can improve modelling accuracy and topic quality, especially in the case where the word-occurrence information in the training data is insufficient. In this paper, we present a topic model, called MetaLDA, which is able to leverage either document or word meta information, or both of them jointly. With two data argumentation techniques, we can derive an efficient Gibbs sampling algorithm, which benefits from the fully local conjugacy of the model. Moreover, the algorithm is favoured by the sparsity of the meta information. Extensive experiments on several real world datasets demonstrate that our model achieves comparable or improved performance in terms of both perplexity and topic quality, particularly in handling sparse texts. In addition, compared with other models using meta information, our model runs significantly faster.

Motivation & Objective

• To address the limitations of conventional topic models that ignore rich meta information associated with documents and words.
• To improve topic modeling accuracy and coherence, especially in low-word-co-occurrence scenarios like short texts.
• To develop a scalable and efficient inference algorithm that leverages both document-level and word-level meta information simultaneously.
• To achieve faster training than existing meta-information-augmented topic models without sacrificing performance.
• To enable flexible integration of diverse meta information types (binary or real-valued) into topic modeling frameworks.

Proposed method

• MetaLDA uses data augmentation techniques to achieve full local conjugacy, enabling closed-form Gibbs sampling for efficient inference.
• Document meta information is incorporated via topic distribution priors: documents with similar labels share similar Dirichlet priors for their topic distributions.
• Word meta information is integrated into the per-topic word distribution priors, encouraging semantically similar words to have similar topic weights.
• The model supports joint use of document and word meta information, with separate inference procedures for each modality.
• A sparse caching strategy is applied to reduce computational complexity, especially when using document-level meta information.
• The implementation leverages parallelization via the DistributedLDA framework for scalability on large datasets.

Experimental results

Research questions

• RQ1 Can incorporating both document-level and word-level meta information jointly improve topic model performance on sparse texts?
• RQ2 How does MetaLDA’s inference efficiency compare to existing meta-information-augmented topic models?
• RQ3 Does the use of meta information lead to better topic coherence and lower perplexity, especially when word co-occurrence data is limited?
• RQ4 Can the model maintain high performance while achieving significantly faster inference than comparable models?
• RQ5 How does the model scale with increasing numbers of topics and large-scale datasets?

Key findings

• MetaLDA achieved the highest NPMI scores on the WS and AN datasets, indicating superior topic coherence, particularly after removing low-quality topics.
• On the TMN dataset, MetaLDA’s NPMI score was the best on average, though not statistically distinct from others due to overlapping standard deviations.
• MetaLDA was 6–8 times faster than DMR on the Reuters dataset and over 10 times faster on the WS dataset when using only document meta information.
• With word meta information, MetaLDA was 20–30 times faster than WF-LDA and LF-LDA on the WS dataset, especially as the number of topics increased.
• On the large NYT dataset, MetaLDA maintained its speed advantage in parallel settings, outperforming DMR, LF-LDA, and WF-LDA in per-iteration running time.
• MetaLDA achieved lower perplexity and better topic quality than baseline models, particularly in low-data regimes such as short texts.