[Paper Review] Keyword Assisted Topic Models

Shusei Eshima, Kosuke Imai+1 more|arXiv (Cornell University)|Apr 13, 2020

Computational and Text Analysis Methods52 references33 citations

TL;DR

The paper introduces keyATM, a semi-supervised topic model that uses a small set of keywords to improve interpretability and measurement, and supports no-keyword topics, covariates, and time trends.

ABSTRACT

In recent years, fully automated content analysis based on probabilistic topic models has become popular among social scientists because of their scalability. The unsupervised nature of the models makes them suitable for exploring topics in a corpus without prior knowledge. However, researchers find that these models often fail to measure specific concepts of substantive interest by inadvertently creating multiple topics with similar content and combining distinct themes into a single topic. In this paper, we empirically demonstrate that providing a small number of keywords can substantially enhance the measurement performance of topic models. An important advantage of the proposed keyword assisted topic model (keyATM) is that the specification of keywords requires researchers to label topics prior to fitting a model to the data. This contrasts with a widespread practice of post-hoc topic interpretation and adjustments that compromises the objectivity of empirical findings. In our application, we find that keyATM provides more interpretable results, has better document classification performance, and is less sensitive to the number of topics than the standard topic models. Finally, we show that keyATM can also incorporate covariates and model time trends. An open-source software package is available for implementing the proposed methodology.

Motivation & Objective

• Motivate the need for improved measurement in automated content analysis using topic models.
• Propose a semi-supervised topic model (keyATM) that incorporates a small set of keywords for each topic.
• Extend the base model to allow topics without keywords and to model document covariates and time trends.
• Demonstrate that keyword incorporation yields more interpretable topics and better classification performance than unsupervised baselines.

Proposed method

• Define two-topic-type structure: keyword topics and no-keyword topics within a K-topic model.
• For keyword topics, introduce a Bernoulli s_di to decide if a word comes from keywords or the standard topic-word distribution.
• Use Dirichlet priors for topic-word and keyword-word distributions, with a Beta prior on the keyword probability parameter pi_k.
• Employ a collapsed Gibbs sampling scheme to sample z_di, s_di, and alpha_k, integrating out theta, phi, tilde_phi, and pi.
• Incorporate term weighting (wLDA) to downweight overly frequent words in counts during sampling.
• Provide closed-form expressions for posterior estimates of phi*_kv and theta_dk and discuss interpretation of keyword and non-keyword components.

Experimental results

Research questions

• RQ1 Does incorporating a small set of keywords for topics improve interpretability of topics compared to unsupervised topic models?
• RQ2 Does keyATM yield better document classification performance against standard LDA-based models?
• RQ3 Can keyATM accommodate topics with no keywords and model covariates/time trends without sacrificing performance?

Key findings

• keyATM yields more interpretable topic-word distributions than a keyword-unaware baseline (wLDA).
• Topic-word distributions from keyATM align better with human-coded labels and CAP/CBP classifications.
• keyATM achieves better document-topic classification performance than wLDA across most topics in the Congressional bills corpus, with ROC comparisons favoring keyATM.
• Allowing no-keyword topics and learning hyper-parameters improves model flexibility and performance.
• The base keyATM can incorporate covariates and model time trends, while maintaining improved interpretability and measurement quality.