[Paper Review] CuLDA_CGS: Solving Large-scale LDA Problems on GPUs

Xiaolong Xie, Yun Liang+2 more|arXiv (Cornell University)|Mar 13, 2018

Advanced Neural Network Applications32 references3 citations

TL;DR

CuLDA_CGS is a GPU-accelerated, highly scalable Latent Dirichlet Allocation (LDA) framework that optimizes workload partitioning, memory access, and multi-GPU synchronization to achieve up to 7.3X speedup over state-of-the-art CPU and GPU LDA solutions on a single GPU, with an additional 3.0X speedup on four GPUs, significantly improving throughput for large-scale topic modeling.

ABSTRACT

Latent Dirichlet Allocation(LDA) is a popular topic model. Given the fact that the input corpus of LDA algorithms consists of millions to billions of tokens, the LDA training process is very time-consuming, which may prevent the usage of LDA in many scenarios, e.g., online service. GPUs have benefited modern machine learning algorithms and big data analysis as they can provide high memory bandwidth and computation power. Therefore, many frameworks, e.g. Ten- sorFlow, Caffe, CNTK, support to use GPUs for accelerating the popular machine learning data-intensive algorithms. However, we observe that LDA solutions on GPUs are not satisfying. In this paper, we present CuLDA_CGS, a GPU-based efficient and scalable approach to accelerate large-scale LDA problems. CuLDA_CGS is designed to efficiently solve LDA problems at high throughput. To it, we first delicately design workload partition and synchronization mechanism to exploit the benefits of mul- tiple GPUs. Then, we offload the LDA sampling process to each individual GPU by optimizing from the sampling algorithm, par- allelization, and data compression perspectives. Evaluations show that compared with state-of-the-art LDA solutions, CuLDA_CGS outperforms them by a large margin (up to 7.3X) on a single GPU. CuLDA_CGS is able to achieve extra 3.0X speedup on 4 GPUs. The source code is publicly available on https://github.com/cuMF/ CuLDA_CGS.

Motivation & Objective

• Address the performance bottleneck in large-scale LDA training caused by memory bandwidth limitations on CPUs and distributed systems.
• Overcome the inefficiency of existing GPU-based LDA solutions that fail to fully utilize GPU hardware due to irregular memory access and poor multi-GPU scaling.
• Design a system that achieves high throughput and strong scalability on multi-GPU platforms using a single-machine architecture.
• Optimize the LDA sampling process through algorithmic, parallelization, and data compression techniques to minimize memory footprint and maximize GPU utilization.

Proposed method

• Employ a custom workload partitioning strategy that balances computation across multiple GPUs while minimizing inter-GPU communication.
• Implement a fine-grained synchronization mechanism to reduce overhead during model updates across GPUs.
• Optimize the collapsed Gibbs sampling algorithm by reusing sub-expressions and compressing data structures to reduce memory bandwidth pressure.
• Leverage software-managed caching and memory access patterns to improve cache efficiency and reduce global memory transactions.
• Design a multi-GPU data layout that enables efficient parallel sampling and model aggregation with minimal synchronization latency.
• Utilize CUDA kernels to fully exploit the massive parallelism of GPU architectures, especially for the O(K) per-token sampling step in LDA.

Experimental results

Research questions

• RQ1 How can LDA training be efficiently mapped to GPU architectures to overcome memory bandwidth bottlenecks?
• RQ2 What workload partitioning and synchronization strategies enable high scalability across multiple GPUs in LDA training?
• RQ3 Can a single-machine, multi-GPU LDA system outperform distributed CPU-based systems in terms of throughput and convergence speed?
• RQ4 How does the proposed optimization stack (algorithmic, parallelization, data layout) compare to existing GPU and CPU-based LDA solutions in real-world workloads?
• RQ5 To what extent can CuLDA_CGS scale across different GPU generations and varying numbers of GPUs?

Key findings

• CuLDA_CGS achieves up to 7.3X higher sampling speed than the state-of-the-art CPU-based WarpLDA across all evaluated platforms and datasets.
• On a single Titan X GPU, CuLDA_CGS processes 173.6M tokens per second on the NYTimes dataset, outperforming SaberLDA (120M tokens/sec) on a more powerful GTX 1080 GPU.
• CuLDA_CGS scales efficiently across multiple GPUs, achieving 2.99X speedup on four GPUs compared to a single GPU, demonstrating strong multi-GPU scalability.
• The system maintains high performance across different GPU architectures, including Volta and Pascal, indicating cross-platform portability and optimization.
• By minimizing memory footprint through data compression and sub-expression reuse, CuLDA_CGS reduces memory bandwidth pressure, making it more efficient than cache-optimized CPU solutions.
• The performance advantage of CuLDA_CGS is attributed to its efficient use of GPU memory bandwidth and reduced synchronization overhead, outperforming network-limited distributed LDA systems.