[Paper Review] Embedding Words as Distributions with a Bayesian Skip-gram Model

Arthur Bražinskas, Serhii Havrylov+1 more|arXiv (Cornell University)|Nov 29, 2017

Topic Modeling6 references21 citations

TL;DR

This paper proposes a Bayesian Skip-gram model that represents words as probability density distributions rather than fixed vectors, enabling context-dependent uncertainty modeling through approximate posterior inference. Unlike prior methods, it generates word embeddings as distributions from word-specific priors and yields context-sensitive posterior densities, demonstrating strong performance on lexical substitution and standard NLP benchmarks using a variational autoencoding framework.

ABSTRACT

We introduce a method for embedding words as probability densities in a low-dimensional space. Rather than assuming that a word embedding is fixed across the entire text collection, as in standard word embedding methods, in our Bayesian model we generate it from a word-specific prior density for each occurrence of a given word. Intuitively, for each word, the prior density encodes the distribution of its potential 'meanings'. These prior densities are conceptually similar to Gaussian embeddings. Interestingly, unlike the Gaussian embeddings, we can also obtain context-specific densities: they encode uncertainty about the sense of a word given its context and correspond to posterior distributions within our model. The context-dependent densities have many potential applications: for example, we show that they can be directly used in the lexical substitution task. We describe an effective estimation method based on the variational autoencoding framework. We also demonstrate that our embeddings achieve competitive results on standard benchmarks.

Motivation & Objective

• To address the limitation of fixed word embeddings in capturing polysemy and context-dependent meaning.
• To model word meanings not as single vectors but as distributions that reflect uncertainty and sense variation.
• To enable context-specific inference by deriving posterior densities from a generative Bayesian framework.
• To demonstrate the utility of these context-sensitive distributions in downstream NLP tasks like lexical substitution.

Proposed method

• The model uses a Bayesian extension of the skip-gram framework, where each word occurrence draws its latent meaning from a word-specific prior distribution.
• It employs a variational autoencoding (VAE) framework, with the encoder producing context-specific posterior distributions over word meanings.
• The prior distributions are parameterized per word and serve as data-dependent priors, encoding potential meanings.
• Posterior distributions are inferred via variational inference, allowing uncertainty modeling for each word in context.
• The model optimizes a variational lower bound (ELBO) using stochastic gradient descent with negative sampling.
• Spherical covariance matrices are used for both prior and posterior distributions, improving training stability and performance.

Experimental results

Research questions

• RQ1 Can word embeddings be effectively modeled as probability densities to capture polysemy and semantic uncertainty?
• RQ2 Can context-specific posterior distributions improve performance on lexical substitution and other NLP tasks?
• RQ3 How do context-sensitive posterior densities compare to fixed-distribution embeddings in modeling word meaning?
• RQ4 Does modeling word meaning as distributions enable better capture of semantic relations like entailment?

Key findings

• The Bayesian Skip-gram model successfully generates context-sensitive posterior distributions that reflect the sense of a word in context, such as shifting toward 'bird' when 'kiwi' appears in a bird-related context.
• The model achieves state-of-the-art performance on the lexical substitution task, demonstrating the practical utility of posterior densities for word replacement prediction.
• Context-specific posterior distributions outperform fixed-distribution embeddings in capturing sense variation, particularly for polysemous words.
• The model shows strong performance on standard word similarity and analogy benchmarks, with results competitive with skip-gram and Gaussian embedding (W2G) baselines.
• Spherical covariance matrices yield better performance than diagonal or full covariances, suggesting that isotropic uncertainty is sufficient for context-specific modeling.
• The model's ability to represent entailment via level set inclusion is validated, though direct KL divergence usage is found problematic for entailment detection.