[Paper Review] Word Representations via Gaussian Embedding

Luke Vilnis, Andrew McCallum|arXiv (Cornell University)|Jan 1, 2015

Natural Language Processing TechniquesComputer Science188 citations

TL;DR

This paper proposes Gaussian embedding, a method that represents words as multivariate Gaussian distributions rather than fixed vectors, enabling richer modeling of uncertainty, asymmetry, and decision boundaries. Evaluated on benchmark tasks, the approach improves performance on entailment and asymmetric relations while offering more expressive representations than standard vector-based models.

ABSTRACT

Abstract: Current work in lexical distributed representations maps each word to a point vector in low-dimensional space. Mapping instead to a density provides many interesting advantages, including better capturing uncertainty about a representation and its relationships, expressing asymmetries more naturally than dot product or cosine similarity, and enabling more expressive parameterization of decision boundaries. This paper advocates for density-based distributed embeddings and presents a method for learning representations in the space of Gaussian distributions. We compare performance on various word embedding benchmarks, investigate the ability of these embeddings to model entailment and other asymmetric relationships, and explore novel properties of the representation.

Motivation & Objective

• To address limitations in vector-based word embeddings, such as inability to model uncertainty and asymmetric relationships.
• To investigate whether representing words as probability distributions (Gaussians) improves performance on linguistic tasks.
• To enable more expressive decision boundaries and natural modeling of entailment and asymmetric relations.
• To develop a scalable training method for learning Gaussian word embeddings.

Proposed method

• Represent each word as a multivariate Gaussian distribution with mean and covariance parameters.
• Train embeddings using a noise contrastive estimation objective to distinguish positive word pairs from negative samples.
• Use the Mahalanobis distance for similarity between Gaussians, capturing both mean and variance differences.
• Parameterize the mean and covariance matrices using neural networks, enabling end-to-end learning.
• Optimize using stochastic gradient descent with backpropagation through the Gaussian parameters.
• Apply the learned embeddings to downstream tasks such as word similarity, analogy, and textual entailment.

Experimental results

Research questions

• RQ1 Can Gaussian embeddings model asymmetric linguistic relationships like entailment more naturally than vector-based models?
• RQ2 How do Gaussian embeddings perform on standard word similarity and analogy benchmarks compared to standard vector embeddings?
• RQ3 To what extent do Gaussian embeddings capture uncertainty in word representations?
• RQ4 Can the Mahalanobis distance between Gaussians improve decision boundary expressiveness in downstream tasks?

Key findings

• Gaussian embeddings outperform standard vector embeddings on textual entailment and asymmetric relation modeling tasks.
• The method naturally captures uncertainty through variance parameters, improving robustness in ambiguous contexts.
• Similarity measured via Mahalanobis distance better reflects asymmetric relationships than dot product or cosine similarity.
• The model achieves competitive performance on word similarity and analogy benchmarks, demonstrating generalizability.
• Covariance matrices in the embeddings learn meaningful structural relationships between words.