[Paper Review] word2vec Parameter Learning Explained

Xin Rong|arXiv (Cornell University)|Nov 11, 2014

Topic ModelingComputer Science5 references659 citations

TL;DR

This paper provides a comprehensive mathematical derivation and intuitive explanation of the parameter learning process in word2vec models, covering both continuous bag-of-words (CBOW) and skip-gram architectures, with detailed derivations of gradient updates using stochastic gradient descent, hierarchical softmax, and negative sampling. The key contribution is a clear, step-by-step explanation of how word vector representations are learned through backpropagation and optimization techniques, making the inner workings of word2vec accessible to non-experts in neural networks.

ABSTRACT

The word2vec model and application by Mikolov et al. have attracted a great amount of attention in recent two years. The vector representations of words learned by word2vec models have been shown to carry semantic meanings and are useful in various NLP tasks. As an increasing number of researchers would like to experiment with word2vec or similar techniques, I notice that there lacks a material that comprehensively explains the parameter learning process of word embedding models in details, thus preventing researchers that are non-experts in neural networks from understanding the working mechanism of such models. This note provides detailed derivations and explanations of the parameter update equations of the word2vec models, including the original continuous bag-of-word (CBOW) and skip-gram (SG) models, as well as advanced optimization techniques, including hierarchical softmax and negative sampling. Intuitive interpretations of the gradient equations are also provided alongside mathematical derivations. In the appendix, a review on the basics of neuron networks and backpropagation is provided. I also created an interactive demo, wevi, to facilitate the intuitive understanding of the model.

Motivation & Objective

• To provide a detailed, accessible explanation of the parameter learning process in word2vec models for researchers without advanced neural network expertise.
• To derive and explain the gradient update equations for both CBOW and skip-gram models using stochastic gradient descent.
• To clarify the mathematical foundations of advanced optimization techniques such as hierarchical softmax and negative sampling in word2vec.
• To bridge the gap between intuitive understanding and formal derivation of word embedding training via backpropagation.
• To support learning with an interactive demo (wevi) and a review of neural network basics in the appendix.

Proposed method

• Derives the loss function for word2vec as the negative log-likelihood of predicting the correct target word given a context.
• Derives the gradient of the loss with respect to the output layer weights using backpropagation, resulting in the update rule: $ \mathbf{v}'_{w_j}^{\text{new}} = \mathbf{v}'_{w_j}^{\text{old}} - \eta (y_j - t_j) \mathbf{h} $.
• Applies the same derivation to the input vectors by backpropagating the error through the hidden layer to obtain the update for $ \mathbf{v}_w $.
• Introduces and derives the negative sampling method as a computationally efficient alternative to full softmax, using a noise distribution $ P_n(w) $ to sample negative words.
• Derives the loss function for negative sampling: $ E = -\log\sigma(\mathbf{v}'_{w_O}^T \mathbf{h}) - \sum_{w_j \in \mathcal{W}_{\text{neg}}} \log\sigma(-\mathbf{v}'_{w_j}^T \mathbf{h}) $.
• Derives the gradient update for both output and input vectors under negative sampling: $ \mathbf{v}'_{w_j}^{\text{new}} = \mathbf{v}'_{w_j}^{\text{old}} - \eta (\sigma(\mathbf{v}'_{w_j}^T \mathbf{h}) - t_j) \mathbf{h} $, applied only to positive and sampled negative words.

Experimental results

Research questions

• RQ1 How are the word vector parameters updated during training in the CBOW and skip-gram models?
• RQ2 What is the mathematical derivation of the gradient update rule for the output layer weights in the standard word2vec model with softmax?
• RQ3 How does negative sampling reduce computational cost while preserving effective word vector learning?
• RQ4 What is the role of the noise distribution $ P_n(w) $ in negative sampling, and how does it affect the training objective?
• RQ5 How does backpropagation propagate the error from the output layer to the input layer to update the input word vectors?

Key findings

• The gradient of the loss with respect to the output vector $ \mathbf{v}'_{w_j} $ is $ (\sigma(\mathbf{v}'_{w_j}^T \mathbf{h}) - t_j) \mathbf{h} $, where $ t_j = 1 $ if $ w_j $ is the correct output word and 0 otherwise.
• The update rule for the output vectors under negative sampling applies only to the positive word and the K sampled negative words, significantly reducing computational cost compared to full softmax.
• The hidden layer output gradient $ \partial E / \partial \mathbf{h} $ is the sum of the output vector gradients weighted by their prediction errors, enabling backpropagation to input vectors.
• The input vector update for CBOW is derived by summing the hidden layer error across all context words, resulting in $ \mathbf{v}_{w_c}^{\text{new}} = \mathbf{v}_{w_c}^{\text{old}} - \eta \cdot \text{EH} / C $, where EH is the total error from the output layer.
• The paper confirms that negative sampling produces high-quality word embeddings with significantly reduced training time, as shown empirically in prior work (Mikolov et al., 2013b).
• The derivation shows that the negative sampling objective function is equivalent to minimizing a contrastive loss that encourages correct words to have higher similarity scores than negative samples.