[Paper Review] Short Text Topic Modeling Techniques, Applications, and Performance: A Survey

Jipeng Qiang, Qian Zhenyu+3 more|arXiv (Cornell University)|Apr 13, 2019

Topic Modeling21 citations

TL;DR

This survey presents a comprehensive analysis of short text topic modeling (STTM) techniques, categorizing them into Dirichlet multinomial mixture (DMM)-based, global word co-occurrence-based, and self-aggregation-based methods. It evaluates state-of-the-art models on real-world datasets, introduces the open-source STTM library in Java, and demonstrates that DMM-based models achieve faster convergence and better efficiency than alternatives, with GPU-accelerated variants showing improved scalability while maintaining topic coherence.

ABSTRACT

Analyzing short texts infers discriminative and coherent latent topics that is a critical and fundamental task since many real-world applications require semantic understanding of short texts. Traditional long text topic modeling algorithms (e.g., PLSA and LDA) based on word co-occurrences cannot solve this problem very well since only very limited word co-occurrence information is available in short texts. Therefore, short text topic modeling has already attracted much attention from the machine learning research community in recent years, which aims at overcoming the problem of sparseness in short texts. In this survey, we conduct a comprehensive review of various short text topic modeling techniques proposed in the literature. We present three categories of methods based on Dirichlet multinomial mixture, global word co-occurrences, and self-aggregation, with example of representative approaches in each category and analysis of their performance on various tasks. We develop the first comprehensive open-source library, called STTM, for use in Java that integrates all surveyed algorithms within a unified interface, benchmark datasets, to facilitate the expansion of new methods in this research field. Finally, we evaluate these state-of-the-art methods on many real-world datasets and compare their performance against one another and versus long text topic modeling algorithm.

Motivation & Objective

• To provide a systematic taxonomy of short text topic modeling (STTM) techniques addressing the sparsity problem in short texts.
• To evaluate the performance of representative STTM methods across multiple real-world datasets and tasks such as classification, clustering, and topic coherence.
• To develop the first open-source, unified Java library (STTM) integrating surveyed algorithms, benchmark datasets, and evaluation tools for future research.
• To compare the efficiency, convergence speed, and topic quality of DMM-based, global co-occurrence-based, and self-aggregation-based models.
• To identify open challenges in visualization, evaluation metrics, and model selection for STTM in real-world applications.

Proposed method

• Categorizes STTM methods into three groups: DMM-based (e.g., GSDMM, GPU-DMM), global word co-occurrence-based (e.g., BTM, WNTM), and self-aggregation-based (e.g., PTM, SATM).
• Employs variational inference and Gibbs sampling for parameter estimation in DMM and related models, with GPU acceleration for large-scale inference.
• Introduces a unified open-source Java library (STTM) that provides a common interface for all surveyed models and supports evaluation tasks like classification, clustering, and coherence scoring.
• Uses six public datasets (e.g., Biomedicine, GoogleNews, Tweet) to benchmark model performance across topic coherence (NMI), convergence speed, and runtime efficiency.
• Applies standard evaluation metrics including Normalized Mutual Information (NMI) and topic coherence to compare models across different corpora.
• Analyzes convergence behavior by measuring NMI values over iterations, revealing faster convergence for DMM-based models compared to self-aggregation-based ones.

Experimental results

Research questions

• RQ1 How do different categories of short text topic modeling methods—DMM-based, global co-occurrence-based, and self-aggregation-based—perform in terms of topic coherence and convergence speed?
• RQ2 What are the efficiency trade-offs between GPU-accelerated models and CPU-based counterparts in short text topic modeling?
• RQ3 How do STTM models compare to traditional long-text topic models like LDA in performance on short text corpora?
• RQ4 Which model architecture achieves the best balance between topic quality, convergence speed, and computational efficiency on real-world short text datasets?
• RQ5 What are the key challenges in evaluating topic models, and how can new metrics and visualization techniques improve interpretability and model selection?

Key findings

• DMM-based models such as GSDMM and GPU-DMM converge fastest, achieving stable performance within 30 iterations, while self-aggregation-based models like SATM show the slowest convergence and lowest iterative performance.
• LDA and DMM are the most efficient models in terms of both initialization and per-iteration time, with LDA requiring 77 ms and DMM 46 ms on the Biomedicine dataset.
• GPU-PDMM is the slowest model, with a per-iteration time of 9685.44 ms on the Biomedicine dataset, due to high computational costs in sampling topic assignments.
• Models incorporating word embeddings (e.g., GPU-DMM, LF-DMM, GPU-PDMM) incur significantly higher initialization times due to similarity computations between words.
• Global word co-occurrence-based models are slower than GSDMM and LDA but faster than self-aggregation-based models, with PTM being faster than SATM but slower than co-occurrence-based methods.
• NMI values stabilize earlier for DMM-based models (within 30 iterations) compared to global co-occurrence-based models (within 60 iterations), confirming faster convergence.