[Paper Review] Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems

Weijie Zhao, Deping Xie+5 more|arXiv (Cornell University)|Mar 12, 2020

Advanced Image and Video Retrieval Techniques49 references76 citations

TL;DR

The paper introduces a distributed hierarchical GPU parameter server (HBM-PS MEM-PS SSD-PS) to train terabyte-scale sparse CTR models, achieving 1.8–4.8x faster training and 4–9x better price-performance than an MPI cluster.

ABSTRACT

Neural networks of ads systems usually take input from multiple resources, e.g., query-ad relevance, ad features and user portraits. These inputs are encoded into one-hot or multi-hot binary features, with typically only a tiny fraction of nonzero feature values per example. Deep learning models in online advertising industries can have terabyte-scale parameters that do not fit in the GPU memory nor the CPU main memory on a computing node. For example, a sponsored online advertising system can contain more than $10^{11}$ sparse features, making the neural network a massive model with around 10 TB parameters. In this paper, we introduce a distributed GPU hierarchical parameter server for massive scale deep learning ads systems. We propose a hierarchical workflow that utilizes GPU High-Bandwidth Memory, CPU main memory and SSD as 3-layer hierarchical storage. All the neural network training computations are contained in GPUs. Extensive experiments on real-world data confirm the effectiveness and the scalability of the proposed system. A 4-node hierarchical GPU parameter server can train a model more than 2X faster than a 150-node in-memory distributed parameter server in an MPI cluster. In addition, the price-performance ratio of our proposed system is 4-9 times better than an MPI-cluster solution.

Motivation & Objective

• Motivate the need to train ultra-large CTR models that exceed GPU memory and CPU memory on a single node.
• Propose a three-tier hierarchical storage design (HBM, memory, SSD) to enable GPU-centered training of massive sparse models.
• Develop efficient intra- and inter-node GPU parameter synchronization to accelerate training.
• Evaluate scalability and compare against a standard MPI cluster baseline in real ads datasets.

Proposed method

• Design a four-stage pipeline to overlap data transfer, parameter loading, and GPU computation.
• Implement a multi-GPU distributed hash table in HBM across GPUs to store working parameters with atomic updates.
• Use RDMA for inter-node GPU parameter synchronization via all-reduce operations.
• Cluster parameters into files on SSDs and employ a file-level parameter management plus background compaction to manage stale data.
• Partition parameters across GPUs and nodes using modulo hashing to map keys to storage locations.

Experimental results

Research questions

• RQ1 Can a hierarchical GPU parameter server enable terabyte-scale CTR models to train efficiently without sacrificing accuracy?
• RQ2 What are the performance and cost benefits of integrating HBM-PS, MEM-PS, and SSD-PS compared with traditional MPI-based training?
• RQ3 How do data transfer, caching, and I/O strategies impact overall training throughput on real-world ads data?

Key findings

• The 4-node hierarchical GPU parameter server achieves 1.8–4.8x training speedup over an MPI-cluster baseline across five CTR models.
• Cost-normalized speedup ranges from 4.4x to 9.0x compared with the MPI solution.
• Relative AUC accuracy of the hierarchical system is within 0.1% of the MPI baseline, with some models slightly outperforming it, indicating lossless training.
• HBM-PS shows that pull/push operations scale with the number of nonzero features, while training time scales with dense parameter count.
• MEM-PS and SSD-PS reduce SSD I/O impact via caching and file-level parameter management, enabling out-of-main-memory training.
• Experiments use 4 GPU nodes (8×32 GB HBMs per node) and 5 CTR models with sparse parameters ranging from 8e9 to 1e11, demonstrating scalability and efficiency.