[Paper Review] Dynamic and Static Topic Model for Analyzing Time-Series Document Collections

Rem Hida, Naoya Takeishi+2 more|arXiv (Cornell University)|May 6, 2018

Advanced Text Analysis Techniques8 references3 citations

TL;DR

This paper proposes a Dynamic and Static Topic Model (DSTM) that jointly models temporal evolution of topics and hierarchical topic dependencies in time-series document collections. By integrating dynamic topic transitions from multiple past topics and static topic hierarchies, DSTM achieves superior performance in perplexity and reveals interpretable topic structures, outperforming LDA, PAM, and DRTM on scientific paper datasets.

ABSTRACT

For extracting meaningful topics from texts, their structures should be considered properly. In this paper, we aim to analyze structured time-series documents such as a collection of news articles and a series of scientific papers, wherein topics evolve along time depending on multiple topics in the past and are also related to each other at each time. To this end, we propose a dynamic and static topic model, which simultaneously considers the dynamic structures of the temporal topic evolution and the static structures of the topic hierarchy at each time. We show the results of experiments on collections of scientific papers, in which the proposed method outperformed conventional models. Moreover, we show an example of extracted topic structures, which we found helpful for analyzing research activities.

Motivation & Objective

• To address the limitation of existing topic models that consider either dynamic or static topic structures, but not both simultaneously.
• To improve topic model interpretability and modeling accuracy by incorporating both temporal dependencies and hierarchical topic relationships in document collections.
• To enable better analysis of research trends and topic evolution in scientific literature and news archives.
• To develop a unified probabilistic framework that captures multi-topic dependencies over time and topic hierarchies at each time point.

Proposed method

• DSTM uses a dynamic structure where topic-word distributions at time t are modeled as a weighted sum of topic-word distributions from time t-1, with weights βt capturing dependencies from multiple past topics.
• A static structure is modeled via a hierarchical prior: supertopics at each time point govern subtopics, with a prior αst controlling the strength of subtopic relationships.
• The model employs a generative process that assigns words to subtopics, subtopics to supertopics, and supertopics to documents, with time-dependent topic distributions.
• Inference and learning are performed using variational Bayesian inference, optimizing the variational lower bound on the marginal likelihood.
• The model jointly learns dynamic transition weights βt and static hierarchy weights αst through iterative optimization.
• The method supports both topic modeling and topic structure extraction, enabling visualization of evolving and hierarchical topic relationships.

Experimental results

Research questions

• RQ1 Can a topic model simultaneously capture dynamic topic evolution and static topic hierarchies in time-series document collections?
• RQ2 How does joint modeling of dynamic and static structures improve topic model performance compared to models that consider only one structure?
• RQ3 To what extent can the extracted topic structures reveal meaningful research trends and inter-topic relationships in scientific literature?
• RQ4 Does incorporating multi-topic dependencies in dynamic evolution lead to better perplexity and interpretability than single-topic dependency models?

Key findings

• DSTM achieved the lowest perplexity across all settings on both the NIPS and Drone datasets, outperforming LDA, PAM, and DRTM.
• On the NIPS dataset, DSTM achieved a perplexity of 1378.7 (±16.5) for K=30, compared to 1455.6 (±16.7) for LDA and 1380.7 (±18.5) for DRTM, with p-values < 0.001 in paired t-tests.
• On the Drone dataset, DSTM achieved a perplexity of 1171.6 (±141.4) for K=25, significantly lower than DRTM’s 1201.2 (±143.5) and LDA’s 1644.7 (±193.0).
• The model successfully extracted interpretable topic structures, showing that 'planning' topics evolved to incorporate 'mapping' topics by 2015–2016, reflecting real-world research trends.
• Dynamic relations revealed increasing interest in 'online' planning from 2016, linked to advances in on-board computing.
• Static relations showed that 'planning' was linked to 'hardware' and 'control' in 2013–2014, but to 'mapping' in 2015–2016, indicating shifting research focus.