[Paper Review] Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec

Christopher E. Moody|arXiv (Cornell University)|May 6, 2016

Topic Modeling14 references147 citations

TL;DR

lda2vec jointly learns dense word vectors and sparse, interpretable document-topic mixtures by integrating SGNS with Dirichlet-constrained document weights and shared topic vectors.

ABSTRACT

Distributed dense word vectors have been shown to be effective at capturing token-level semantic and syntactic regularities in language, while topic models can form interpretable representations over documents. In this work, we describe lda2vec, a model that learns dense word vectors jointly with Dirichlet-distributed latent document-level mixtures of topic vectors. In contrast to continuous dense document representations, this formulation produces sparse, interpretable document mixtures through a non-negative simplex constraint. Our method is simple to incorporate into existing automatic differentiation frameworks and allows for unsupervised document representations geared for use by scientists while simultaneously learning word vectors and the linear relationships between them.

Motivation & Objective

• Motivate the integration of dense word representations with sparse, topic-based document representations for interpretability and downstream use by scientists.
• Develop a model that jointly learns word vectors, topic vectors, and per-document topic mixtures within an automatic differentiation framework.
• Preserve semantic regularities in word vectors while obtaining sparse, human-interpretable document-topic proportions.
• Demonstrate the approach on standard text corpora and a large, domain-specific corpus to show coherent topics and meaningful word analogies.

Proposed method

• Extend the Skipgram Negative Sampling (SGNS) objective to include document-wide feature vectors and document-weighted topic mixtures.
• Represent each context as the sum of a word vector and a document vector (c_j = w_j + d_j) to capture both local and document-level signals.
• Constrain document vectors d_j to be a non-negative, simplex-filled mixture of topic vectors t_k via a softmax transform p_jk, enforcing interpretability.
• Impose a Dirichlet-likelihood term L^d with parameter alpha to encourage sparsity in document-topic memberships (alpha ~ n^{-1}).
• Train end-to-end in minibatches using Adam, with L^d scaled to minibatch size, and evaluate topic coherence (C_v) and token similarity (3COSMUL).
• Optionally, initialize word vectors with pretrained embeddings and share word representations across pivot/target roles in SGNS.

Experimental results

Research questions

• RQ1 Can dense word embeddings and sparse document-topic mixtures be learned jointly in a single differentiable model?
• RQ2 Does enforcing a Dirichlet-based sparsity on document-topic weights yield interpretable topics comparable to LDA?
• RQ3 Do the learned topics align with human-interpretable themes and show coherent word associations within specialized corpora?
• RQ4 Can the model capture meaningful linear relationships between tokens (analogies) within domain-specific vocabularies?
• RQ5 Is the approach scalable to large corpora and amenable to standard deep-learning toolchains?

Key findings

• lda2vec yields coherent topics on the Twenty Newsgroups corpus with topic coherences that correlate with human evaluations.
• On Hacker News comments, lda2vec discovers domain-relevant topics and learns token similarities and linear relationships similar to word analogies.
• The model supports interpretable topic mixtures per document via non-negative weights summing to one, enabling inspection of document-topic composition.
• Topic coherence improves under certain settings, e.g., 20 topics with beta = 0.75 achieving an average coherence of 0.567 (vs other configurations).
• The approach is straightforward to implement in automatic differentiation frameworks and can leverage GPUs for speed.
• Sparse document-topic memberships via Dirichlet regularization yield more coherent topics and prevent incoherent topic bases compared to non-sparse settings.