[Paper Review] TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency

Adji Bousso Dieng, Chong Wang+2 more|arXiv (Cornell University)|Nov 5, 2016

Topic Modeling129 citations

TL;DR

TopicRNN combines an RNN with latent topics to model local word order and global semantic context end-to-end, improving perplexity over contextual RNN baselines and enabling unsupervised document features for sentiment analysis.

ABSTRACT

In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence - both semantic and syntactic - but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global underlying semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned end-to-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis on the IMDB movie review dataset and report an error rate of $6.28\%$. This is comparable to the state-of-the-art $5.91\%$ resulting from a semi-supervised approach. Finally, TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.

Motivation & Objective

• Motivate combining local syntactic modeling of RNNs with global semantic structure from topic models.
• Propose an end-to-end TopicRNN framework that jointly learns RNN parameters and a latent topic representation.
• Handle stop words explicitly to separate the influence of global semantics from local syntax.
• Demonstrate improved perplexity on PTB and competitive sentiment results on IMDB without pre-trained topics.
• Show that TopicRNN can yield coherent topics and serve as an unsupervised feature extractor for downstream tasks.

Proposed method

• Define a generative TopicRNN model where a latent topic vector theta is drawn from a Gaussian prior.
• At each step t, compute RNN hidden state h_t from the previous word x_t and h_{t-1}.
• Introduce a stop-word indicator l_t drawn from a Bernoulli whose rate depends on h_t.
• Model p(y_t|h_t, theta, l_t) with a local term v_i^T h_t and, when l_t=0, a global topic bias b_i^T theta; otherwise theta does not affect the output.
• Use a variational inference network q(theta|X_c, W_c) to approximate the posterior over theta, with X_c as the bag-of-words of non-stop words.
• Optimize the ELBO with reparameterization and train end-to-end using Adam, updating the inference network and model jointly.
• Generate predictions by using a point estimate of theta (the mean of q) and marginalizing over l_t; update theta with a sliding window for efficiency.

Experimental results

Research questions

• RQ1 Can latent topics provide global semantic context to an RNN language model without pre-training or externally supplied topic features?
• RQ2 Does TopicRNN improve word prediction perplexity on PTB compared to contextual RNN baselines?
• RQ3 Can TopicRNN produce meaningful topics and serve as an unsupervised feature extractor for sentiment analysis on IMDB?
• RQ4 How does separating global semantic influence (via theta) from local syntax affect model performance and training dynamics?

Key findings

• TopicRNN yields lower per-word perplexity than the contextual RNN baseline across network sizes on PTB.
• The model with 100 neurons and 50 topics achieves competitive perplexity without pre-trained topic features.
• TopicRNN-derived features enable competitive sentiment analysis on IMDB 100K, with an error rate of 6.28%, close to state-of-the-art methods.
• TopicRNN can generate sensible topics and coherent text samples.
• TopicRNN topics and features provide usable unsupervised representations for downstream tasks such as clustering and sentiment analysis.