[Paper Review] Dynamic Word Embeddings

Robert Bamler, Stephan Mandt|arXiv (Cornell University)|Feb 27, 2017

Topic Modeling7 references108 citations

TL;DR

A probabilistic dynamic skip-gram model that learns continuous word embedding trajectories over time using a diffusion prior and variational inference, enabling smoother, time-aware word meaning shifts.

ABSTRACT

We present a probabilistic language model for time-stamped text data which tracks the semantic evolution of individual words over time. The model represents words and contexts by latent trajectories in an embedding space. At each moment in time, the embedding vectors are inferred from a probabilistic version of word2vec [Mikolov et al., 2013]. These embedding vectors are connected in time through a latent diffusion process. We describe two scalable variational inference algorithms--skip-gram smoothing and skip-gram filtering--that allow us to train the model jointly over all times; thus learning on all data while simultaneously allowing word and context vectors to drift. Experimental results on three different corpora demonstrate that our dynamic model infers word embedding trajectories that are more interpretable and lead to higher predictive likelihoods than competing methods that are based on static models trained separately on time slices.

Motivation & Objective

• Capture semantic evolution of individual words over time from time-stamped text.
• Extend word embeddings to sequential data by linking time-specific embeddings with a latent diffusion process.
• Develop scalable inference algorithms (filtering and smoothing) to train across all time steps jointly.
• Demonstrate improved interpretability and predictive likelihoods over static, time-sliced baselines.

Proposed method

• Generalizes skip-gram to a dynamic setting with two embedding vectors per word (u_i and v_j).
• Models time evolution as a diffusion process with Gaussian priors and Ornstein-Uhlenbeck damping.
• Uses two scalable variational inference algorithms: skip-gram filtering and skip-gram smoothing.
• Implements black-box variational inference with reparameterization and structured time-domain variational distributions.
• In smoothing, uses a factorized time-series Gaussian with a tridiagonal precision to capture temporal correlations.
• Evaluates on large corpora (Google Books, State of the Union, Twitter) and compares predictive likelihoods to static baselines.

Experimental results

Research questions

• RQ1 Can word embedding trajectories be learned continuously over time rather than in separate time bins?
• RQ2 Do dynamic embeddings yield smoother trajectories and more interpretable semantic shifts than static, time-sliced models?
• RQ3 Do sharing information across time improve predictive performance on held-out data, especially for smaller time slices?
• RQ4 How do different inference schemes (filtering vs. smoothing) compare in tracking semantic evolution?
• RQ5 What words exhibit the most significant semantic changes when modeled dynamically across long time spans?

Key findings

• Dynamic skip-gram models yield smoother word trajectories than static baselines.
• Dynamic filtering and smoothing achieve higher held-out predictive likelihoods than static models, especially on smaller corpora.
• Embedding trajectories reveal interpretable semantic shifts, such as changes in technologically related words, with gradual movement rather than abrupt jumps.
• Smoothing (DSG-S) generally outperforms filtering (DSG-F) by leveraging information from both past and future time steps.
• The approach enables automatic identification of words with significant semantic drift over long time spans.