[Paper Review] Efficient Parallel Learning of Word2Vec

Jeroen B. P. Vuurens, Carsten Eickhoff+1 more|arXiv (Cornell University)|Jun 24, 2016

Topic Modeling5 references3 citations

TL;DR

This paper proposes a simple caching strategy to improve the efficiency of parallel Word2Vec training by reducing memory access conflicts in shared-memory, multi-threaded implementations. By caching the 31 most frequently updated vectors in the hierarchical softmax tree—especially the root node—training speed increases by up to 4x compared to the original C implementation, with no loss in model accuracy.

ABSTRACT

Since its introduction, Word2Vec and its variants are widely used to learn semantics-preserving representations of words or entities in an embedding space, which can be used to produce state-of-art results for various Natural Language Processing tasks. Existing implementations aim to learn efficiently by running multiple threads in parallel while operating on a single model in shared memory, ignoring incidental memory update collisions. We show that these collisions can degrade the efficiency of parallel learning, and propose a straightforward caching strategy that improves the efficiency by a factor of 4.

Motivation & Objective

• To address the scalability bottleneck in parallel Word2Vec training caused by memory access conflicts in shared-memory, multi-threaded implementations.
• To investigate why existing implementations like the original C version and Gensim degrade in efficiency beyond 8–16 cores.
• To evaluate whether caching frequently updated vectors can reduce memory contention and improve training throughput.
• To demonstrate that a lightweight, cache-based optimization can outperform complex BLAS-optimized approaches in conventional hardware settings.
• To ensure that the caching strategy does not compromise model accuracy, especially in comparison to mini-batch methods that delay updates.

Proposed method

• Identify the most frequently accessed weight vectors in the hierarchical softmax tree, particularly the top-level nodes (e.g., root and top 31 nodes), which are updated most often due to their role in the path of nearly all words.
• Implement a thread-local cache for these frequently accessed vectors to reduce direct access to shared memory during training.
• Delay the write-back of cached vector updates to main memory by a configurable interval (e.g., every 10 words), minimizing concurrent write conflicts.
• Use a standard multi-threaded, shared-memory training loop with the Skip-gram architecture and hierarchical softmax objective.
• Integrate the caching mechanism into a custom C++ implementation (Cython-based, named Cythnn) to benchmark performance against the original Word2Vec C and Gensim implementations.
• Measure performance across varying numbers of cores and cached vector counts to isolate the impact of caching on memory contention.

Experimental results

Research questions

• RQ1 Why does parallel Word2Vec training degrade in efficiency beyond 8 cores in shared-memory implementations?
• RQ2 To what extent do memory access conflicts from concurrent updates of the same vectors degrade training performance?
• RQ3 Can a lightweight caching strategy that delays writes to shared memory significantly improve training speed without sacrificing model accuracy?
• RQ4 How many of the most frequently updated vectors should be cached to achieve optimal performance gains?
• RQ5 Does caching preserve model quality compared to mini-batch approaches that also delay updates?

Key findings

• The original Word2Vec C implementation shows a 40% increase in execution time when scaling from 8 to 24 cores, indicating severe scalability degradation due to memory contention.
• Caching the top 31 nodes in the hierarchical softmax tree reduces training time by up to 3.9x compared to the fastest original C implementation (630s vs. 157s on Wikipedia data).
• Caching just the root node (1 cached vector) reduces execution time by over 40% when using more than 20 cores, confirming that top-level nodes are the primary source of contention.
• The performance gain from caching is most significant beyond 16 cores, where memory bandwidth becomes a bottleneck and contention increases.
• The caching strategy maintains model accuracy—Cythnn with caching achieves 33.57% accuracy on the word analogy test set, comparable to the original Word2Vec (33.73%) and Gensim (35.45%), with no measurable drop in effectiveness.
• The optimal cache update frequency (e.g., every 10 words) has minimal impact on accuracy, indicating that delayed writes do not degrade model quality in practice.