[Paper Review] Improving Negative Sampling for Word Representation using Self-embedded Features

Long Chen, Fajie Yuan+2 more|arXiv (Cornell University)|Oct 26, 2017

Topic Modeling3 citations

TL;DR

This paper proposes an adaptive negative sampling method for word representation learning that improves upon traditional popularity-based sampling by dynamically selecting informative negative samples using multi-dimensional self-embedded features. By leveraging both embedding vectors and the current SGD parameter state, the method reduces gradient vanishing, accelerates convergence, and achieves significant performance gains over baseline word2vec without increasing computational complexity.

ABSTRACT

Although the word-popularity based negative sampler has shown superb performance in the skip-gram model, the theoretical motivation behind oversampling popular (non-observed) words as negative samples is still not well understood. In this paper, we start from an investigation of the gradient vanishing issue in the skipgram model without a proper negative sampler. By performing an insightful analysis from the stochastic gradient descent (SGD) learning perspective, we demonstrate that, both theoretically and intuitively, negative samples with larger inner product scores are more informative than those with lower scores for the SGD learner in terms of both convergence rate and accuracy. Understanding this, we propose an alternative sampling algorithm that dynamically selects informative negative samples during each SGD update. More importantly, the proposed sampler accounts for multi-dimensional self-embedded features during the sampling process, which essentially makes it more effective than the original popularity-based (one-dimensional) sampler. Empirical experiments further verify our observations, and show that our fine-grained samplers gain significant improvement over the existing ones without increasing computational complexity.

Motivation & Objective

• To address the limitations of popularity-based negative sampling in word2vec, which oversamples frequent but often irrelevant words and ignores dynamic parameter states.
• To investigate the theoretical and empirical causes of gradient vanishing in skip-gram models under poor sampling strategies.
• To develop a sampling method that selects high-informative negative samples based on multi-dimensional semantic and syntactic features rather than frequency alone.
• To ensure the proposed method maintains constant amortized runtime, preserving efficiency while improving model accuracy.

Proposed method

• The proposed Adaptive Sampler selects negative samples based on the inner product score between target word embeddings and context word embeddings, favoring higher-scoring (more relevant) candidates.
• It dynamically updates the sampling distribution during each SGD step, incorporating the current state of model parameters to prioritize informative negatives.
• The method integrates multi-dimensional embedding features (semantic and syntactic) into the sampling process, moving beyond one-dimensional popularity scores.
• The algorithm maintains an amortized O(d) runtime per update, where d is the embedding dimension, matching the cost of a single inner product operation.
• The sampler is integrated into both skip-gram and CBOW architectures, replacing the original uniform or popularity-based negative sampling.
• Theoretical analysis from an SGD optimization perspective justifies that higher inner product scores yield more informative gradients, improving convergence and accuracy.

Experimental results

Research questions

• RQ1 Why does popularity-based negative sampling perform well despite oversampling potentially irrelevant words?
• RQ2 How does the gradient vanishing problem manifest in word2vec when using poor negative sampling strategies?
• RQ3 Can sampling based on embedding similarity (inner product scores) outperform frequency-based sampling in terms of convergence and accuracy?
• RQ4 Does incorporating multi-dimensional semantic features into negative sampling improve word representation quality without increasing computational cost?

Key findings

• The Adaptive Sampler significantly outperforms the original word2vec baseline on word analogy and word similarity tasks, with SGA achieving higher accuracy than SG and SGU.
• On the NewsIR dataset, the method converged when k > 15 for negative samples, indicating robustness and reduced overfitting compared to uniform sampling.
• The CBOWA and SGA models consistently outperformed their counterparts (CBOW and SG), confirming the effectiveness of adaptive sampling across architectures.
• Training time for the Adaptive Sampler increased marginally—e.g., from 37.4 to 388.9 minutes for d=300 on NewsIR—confirming amortized O(d) runtime and no significant computational overhead.
• The method effectively alleviates gradient vanishing by prioritizing high-scoring, semantically relevant negatives, leading to more effective parameter updates.