[Paper Review] Discovering Discrete Latent Topics with Neural Variational Inference

Yishu Miao, Edward Grefenstette+1 more|arXiv (Cornell University)|Jun 1, 2017

Topic ModelingComputer Science25 references162 citations

TL;DR

The paper introduces neural topic models parameterised by Gaussian-based neural networks (GSM, GSB, RSB) and a truncation-free recurrent stick-breaking approach (RSB-TF) to discover discrete latent topics, achieving strong perplexity and topic coherence on multiple corpora.

ABSTRACT

Topic models have been widely explored as probabilistic generative models of documents. Traditional inference methods have sought closed-form derivations for updating the models, however as the expressiveness of these models grows, so does the difficulty of performing fast and accurate inference over their parameters. This paper presents alternative neural approaches to topic modelling by providing parameterisable distributions over topics which permit training by backpropagation in the framework of neural variational inference. In addition, with the help of a stick-breaking construction, we propose a recurrent network that is able to discover a notionally unbounded number of topics, analogous to Bayesian non-parametric topic models. Experimental results on the MXM Song Lyrics, 20NewsGroups and Reuters News datasets demonstrate the effectiveness and efficiency of these neural topic models.

Motivation & Objective

• Motivate neural alternatives to traditional topic models to handle non-conjugate and context-rich document data.
• Propose parameterisations of topic distributions via neural networks that enable end-to-end backpropagation.
• Explore finite (GSM, GSB) and unbounded (RSB, RSB-TF) topic models to model sparse, topic-aware document representations.
• Demonstrate efficiency and robustness of these models on standard datasets (MXM Song Lyrics, 20NewsGroups, Reuters).

Proposed method

• Define three neural constructions for topic distributions conditioned on a Gaussian latent: Gaussian Softmax (GSM), Gaussian Stick Breaking (GSB), and Recurrent Stick Breaking (RSB).
• Use a variational auto-encoder framework with an inference network q(θ|d) to approximate p(θ|d) and apply reparameterisation for gradient estimation.
• Parameterise β (topic-word distributions) via softmax over word vectors and topic vectors, enabling end-to-end training of Θ (topic/word parameters) and Φ (inference parameters).
• Implement an unbounded topic model by coupling RSB with a Topic RNN to dynamically generate an infinite sequence of topics (RSB, RSB-TF).
• Introduce a truncation-free variational inference scheme where new topics are added if they improve the evidence lower bound by a controlled acceptance threshold γ.
• Compare neural topic models to traditional LDA-based and neural document models, analyzing perplexity and topic coherence.

Experimental results

Research questions

• RQ1 Can neural networks parameterise discrete topic distributions effectively for document modelling?
• RQ2 Do GSM, GSB, and RSB offer advantages over traditional LDA and neural baselines in terms of perplexity and coherence on standard corpora?
• RQ3 Can a recurrent stick-breaking construction yield a truncation-free, unbounded topic model that remains trainable via backpropagation?
• RQ4 What is the impact of implicit topic distributions (document-model variant) on generalisation and coherence compared with explicit topic-word distributions?
• RQ5 How does dynamic topic creation (RSB-TF) affect performance and topic diversity across datasets?

Key findings

• GSM, GSB, and RSB outperform several baselines (LDA, NVLDA) in perplexity on MXM, 20NewsGroups, and Reuters datasets.
• RSB-TF (truncation-free) achieves better perplexity than HDP on the tested datasets, demonstrating the benefit of dynamic topic growth.
• Switching to implicit topic distributions improves generalisation and often yields better perplexity than explicit topic-word distributions across models.
• RSB can outperform GSM and GSB on 20NewsGroups in both 50- and 200-topic settings, likely due to capturing dependencies via sequential stick-breaking.
• Topic coherence (NPMI) is comparable across neural topic models, with earlier topics in stick-breaking order often showing higher coherence.
• RSB-TF dynamically activates topics during training, with convergence typically occurring between 200–300 active topics on 20NewsGroups, and showing insensitivity to initial topic count.