[Paper Review] word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method

Yoav Goldberg, Omer Levy|arXiv (Cornell University)|Feb 15, 2014

Topic Modeling2 references1,299 citations

TL;DR

This paper provides a clear derivation and explanation of Mikolov et al.'s negative-sampling method in word2vec, showing how it efficiently approximates the computationally expensive softmax in skip-gram models by training on positive (true) word-context pairs and negatively sampled (false) pairs. The key contribution is a rigorous, intuitive derivation of the negative-sampling objective function, which enables faster training while preserving semantic similarity in learned word embeddings.

ABSTRACT

The word2vec software of Tomas Mikolov and colleagues (https://code.google.com/p/word2vec/ ) has gained a lot of traction lately, and provides state-of-the-art word embeddings. The learning models behind the software are described in two research papers. We found the description of the models in these papers to be somewhat cryptic and hard to follow. While the motivations and presentation may be obvious to the neural-networks language-modeling crowd, we had to struggle quite a bit to figure out the rationale behind the equations. This note is an attempt to explain equation (4) (negative sampling) in "Distributed Representations of Words and Phrases and their Compositionality" by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean.

Motivation & Objective

• To clarify the often-cryptic derivation of the negative-sampling objective in Mikolov et al.'s word2vec paper, which is essential for understanding the method's underlying logic.
• To bridge the gap between the neural network language modeling literature and the practical implementation of word2vec, particularly the negative-sampling technique.
• To provide a mathematically sound and intuitive explanation of why negative sampling works as an efficient alternative to full softmax in training word embeddings.
• To formalize the connection between the negative-sampling objective and the distributional hypothesis, showing how it encourages similar words to have similar vector representations.

Proposed method

• Derives the negative-sampling objective by modeling the probability that a word-context pair is drawn from the training data, using a logistic regression-style binary classification setup.
• Introduces the use of negative samples—randomly drawn word-context pairs not in the training data—to prevent trivial solutions where all vectors become identical.
• Constructs the optimization objective as the sum of log-likelihoods for positive pairs (log σ(v_c · v_w)) and negative pairs (log σ(–v_c · v_w)), which encourages high similarity for true pairs and low similarity for false ones.
• Explains that negative samples are drawn from a distribution proportional to word frequency raised to the 3/4 power, improving training stability and performance.
• Demonstrates that the method is equivalent to optimizing a binary cross-entropy loss over positive and negative pairs, making it computationally tractable compared to full softmax.
• Highlights that the method is non-convex when both word and context vectors are jointly optimized, but becomes convex if one set of vectors is fixed.

Experimental results

Research questions

• RQ1 How does negative sampling in word2vec approximate the computationally expensive softmax in the skip-gram model?
• RQ2 Why does negative sampling lead to effective word embeddings despite optimizing a different objective than the original skip-gram model?
• RQ3 What is the mathematical derivation of the negative-sampling objective function, and how does it relate to the original word2vec formulation?
• RQ4 How do the sampling strategies for negative examples (e.g., frequency-based sampling with exponent 3/4) affect the quality of learned representations?
• RQ5 What is the theoretical justification for why negative sampling produces word embeddings that capture semantic similarity?

Key findings

• Negative sampling provides a computationally efficient alternative to full softmax in word2vec by replacing the expensive normalization over all contexts with a binary classification objective.
• The method optimizes a binary cross-entropy-like objective that maximizes the similarity (dot product) between true word-context pairs and minimizes it for negative pairs.
• The use of negative samples drawn from a skewed distribution (frequency raised to 3/4 power) improves model performance and reduces noise from overly frequent words.
• Subsampling of frequent words increases the effective context window size, allowing distant but meaningful words to contribute to similarity learning.
• Despite optimizing a different objective than the original skip-gram model, negative sampling produces word embeddings that satisfy the distributional hypothesis, where semantically similar words have similar vector representations.
• The method is non-convex when both word and context vectors are trained jointly, but becomes convex when one set of vectors is held fixed, which helps in understanding its optimization behavior.