[Paper Review] Continuous Semantic Topic Embedding Model Using Variational Autoencoder

Namkyu Jung, Hyeong In Choi|arXiv (Cornell University)|Nov 24, 2017

Topic Modeling22 references3 citations

TL;DR

This paper proposes the Continuous Semantic Topic Embedding Model (CSTEM), a variational autoencoder-based topic model that learns continuous topic and word embeddings in Euclidean space using a Mahalanobis distance function to model semantic relationships. By introducing a global word weight parameter, CSTEM improves topic coherence and enables visualizable, semantically meaningful topic and word vector representations with faster inference than sampling-based methods.

ABSTRACT

This paper proposes the continuous semantic topic embedding model (CSTEM) which finds latent topic variables in documents using continuous semantic distance function between the topics and the words by means of the variational autoencoder(VAE). The semantic distance could be represented by any symmetric bell-shaped geometric distance function on the Euclidean space, for which the Mahalanobis distance is used in this paper. In order for the semantic distance to perform more properly, we newly introduce an additional model parameter for each word to take out the global factor from this distance indicating how likely it occurs regardless of its topic. It certainly improves the problem that the Gaussian distribution which is used in previous topic model with continuous word embedding could not explain the semantic relation correctly and helps to obtain the higher topic coherence. Through the experiments with the dataset of 20 Newsgroup, NIPS papers and CNN/Dailymail corpus, the performance of the recent state-of-the-art models is accomplished by our model as well as generating topic embedding vectors which makes possible to observe where the topic vectors are embedded with the word vectors in the real Euclidean space and how the topics are related each other semantically.

Motivation & Objective

• To develop a continuous topic model that captures semantic relationships between topics and words more accurately than traditional LDA.
• To address the limitation of Gaussian-based topic models in capturing semantic meaning by introducing a semantic distance function in Euclidean space.
• To improve topic coherence by incorporating a global word weight parameter independent of topic assignment.
• To generate continuous, visualizable topic and word embeddings in a shared vector space for semantic analysis.
• To achieve state-of-the-art performance in topic coherence and perplexity with faster training than MCMC-based methods.

Proposed method

• CSTEM uses a variational autoencoder (VAE) framework to jointly infer topic distributions and word embeddings via the reparameterization trick.
• It models word occurrence probability as an exponential function of the inverse Mahalanobis distance between topic and word vectors, capturing semantic similarity.
• A global word weight parameter $ c_i $ is introduced to account for word frequency independent of topic, improving modeling of word occurrence.
• The model optimizes a variational lower bound (ELBO) using Monte Carlo estimation and stochastic gradient descent.
• Topic and word vectors are jointly embedded in a shared 300-dimensional Euclidean space, enabling geometric interpretation of semantic relations.
• Dimensionality reduction via PCA is used to visualize topic and word vectors in 2D, preserving semantic clustering.

Experimental results

Research questions

• RQ1 Can a continuous semantic distance function in Euclidean space better model topic-word relationships than discrete or Gaussian-based models?
• RQ2 Does introducing a global word weight parameter improve topic coherence and model fit beyond standard topic models?
• RQ3 Can the learned topic and word embeddings be visually interpreted to reflect semantic similarity between topics and words?
• RQ4 How does the performance of CSTEM compare to state-of-the-art models in terms of topic coherence and perplexity?
• RQ5 Can the continuous vector space enable generation of new, meaningful topics through interpolation?

Key findings

• CSTEM achieves topic coherence comparable to or slightly better than state-of-the-art models like AVITM and Gaussian LDA, with significantly faster training than sampling-based methods.
• The model outperforms LDA and ProdLDA in topic coherence while maintaining competitive perplexity scores.
• Visualizations using PCA show that semantically similar words cluster together and topics with related themes (e.g., war and world) are geometrically close in the embedding space.
• The global word weight parameter $ c_i $ is found to be essential for modeling word frequency independent of topic, improving model fit.
• Topic vectors are successfully embedded in the same space as word vectors, enabling intuitive interpretation of topic semantics through geometric proximity.
• The model enables infinite generation of custom topics by interpolating in the continuous embedding space, demonstrating its generative capability.