[Paper Review] Parallelizing Word2Vec in Shared and Distributed Memory

Shihao Ji, Nadathur Satish+2 more|arXiv (Cornell University)|Apr 15, 2016

Topic Modeling17 references21 citations

TL;DR

This paper proposes a high-performance, scalable word2vec implementation that replaces memory-bandwidth-intensive vector-vector operations with matrix multiply operations via minibatching and shared negative sampling, enabling efficient utilization of modern multi-core and distributed architectures. The method achieves near-linear scaling across 32 nodes, processing over 110 million words per second with minimal accuracy loss, making it the fastest word2vec implementation to date on the one billion word benchmark.

ABSTRACT

Word2Vec is a widely used algorithm for extracting low-dimensional vector representations of words. It generated considerable excitement in the machine learning and natural language processing (NLP) communities recently due to its exceptional performance in many NLP applications such as named entity recognition, sentiment analysis, machine translation and question answering. State-of-the-art algorithms including those by Mikolov et al. have been parallelized for multi-core CPU architectures but are based on vector-vector operations that are memory-bandwidth intensive and do not efficiently use computational resources. In this paper, we improve reuse of various data structures in the algorithm through the use of minibatching, hence allowing us to express the problem using matrix multiply operations. We also explore different techniques to distribute word2vec computation across nodes in a compute cluster, and demonstrate good strong scalability up to 32 nodes. In combination, these techniques allow us to scale up the computation near linearly across cores and nodes, and process hundreds of millions of words per second, which is the fastest word2vec implementation to the best of our knowledge.

Motivation & Objective

• To address the inefficiency of existing word2vec implementations that rely on memory-bandwidth-intensive vector-vector operations (level-1 BLAS), which underutilize modern multi-core and many-core processors.
• To overcome the inherent sequential dependency in stochastic gradient descent (SGD) by enabling parallel computation without sacrificing convergence or accuracy.
• To design a distributed word2vec system that scales efficiently across multiple nodes with minimal communication overhead and high throughput.
• To maintain predictive performance comparable to the original word2vec while achieving unprecedented training speeds on CPU clusters.

Proposed method

• Replace traditional SGD updates with minibatching, where multiple word pairs are processed together in batches to convert level-1 BLAS operations into level-3 BLAS matrix multiplications, improving data reuse and hardware utilization.
• Use shared negative samples across minibatches to reduce redundant computation and improve model update efficiency.
• Implement a synchronous model update strategy across distributed nodes, with adjustable synchronization frequency to balance communication and computation.
• Apply a simple learning rate adjustment technique to maintain convergence rate as the number of nodes increases, avoiding complex scheduling methods like AdaGrad or RMSProp.
• Optimize communication by reducing model update frequency and using sub-model synchronization to limit network traffic in distributed clusters.
• Leverage vector units and multiply-add instructions in modern CPUs (Intel Broadwell and Knight's Landing) to accelerate matrix operations and improve performance.

Experimental results

Research questions

• RQ1 Can minibatching and shared negative sampling effectively transform word2vec from level-1 to level-3 BLAS operations to improve hardware utilization?
• RQ2 How can word2vec be efficiently parallelized across shared-memory and distributed-memory systems while maintaining convergence and accuracy?
• RQ3 What is the optimal trade-off between model update frequency and communication cost in a distributed word2vec system with increasing node count?
• RQ4 Can a simple learning rate adjustment strategy maintain convergence and predictive performance without complex adaptive optimization techniques?
• RQ5 To what extent can the proposed method achieve near-linear scalability across cores and nodes while processing hundreds of millions of words per second?

Key findings

• The proposed method achieves near-linear scaling on up to 16 Intel Broadwell nodes and 8 Intel Knight's Landing nodes, with throughput increasing from 20M to 110M words per second.
• On 32 Broadwell nodes, the system processes 110 million words per second with only a 1% drop in word similarity accuracy (63.2 vs. 64.0) and 31.1 in word analogy (vs. 32.4), demonstrating high throughput with minimal accuracy loss.
• The implementation on 4 Broadwell nodes matches the performance of BIDMach on 4 NVIDIA Titan-X GPUs (20M words/sec), showing strong CPU-based performance.
• On 16 Knight's Landing nodes, the system achieves 94.7 million words per second, demonstrating high efficiency on many-core architectures.
• Scalability degrades beyond 16 nodes due to increased model synchronization frequency, but even at 32 nodes, the system maintains over 100 million words/sec with acceptable accuracy.
• The method maintains predictive performance comparable to the original word2vec model, with word similarity scores of 63.2 and word analogy scores of 31.1 on 32 Broadwell nodes, confirming that high speed does not compromise model quality.