[Paper Review] Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity

Minghao Yan, Nicholas Meisburger+2 more|arXiv (Cornell University)|Jan 29, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes D-SLIDE, a distributed model-parallel training framework that enables training near-billion-parameter neural networks on low-bandwidth, low-resource CPU clusters by leveraging adaptive sparsity from the SLIDE algorithm. By distributing sparse weight updates and hash tables across nodes, D-SLIDE reduces communication by 99% compared to dense methods, achieving training speeds up to 80x faster than Horovod and matching or exceeding GPU performance on 4–16 core CPUs with 1 Gbps interconnect.

ABSTRACT

More than 70% of cloud computing is paid for but sits idle. A large fraction of these idle compute are cheap CPUs with few cores that are not utilized during the less busy hours. This paper aims to enable those CPU cycles to train heavyweight AI models. Our goal is against mainstream frameworks, which focus on leveraging expensive specialized ultra-high bandwidth interconnect to address the communication bottleneck in distributed neural network training. This paper presents a distributed model-parallel training framework that enables training large neural networks on small CPU clusters with low Internet bandwidth. We build upon the adaptive sparse training framework introduced by the SLIDE algorithm. By carefully deploying sparsity over distributed nodes, we demonstrate several orders of magnitude faster model parallel training than Horovod, the main engine behind most commercial software. We show that with reduced communication, due to sparsity, we can train close to a billion parameter model on simple 4-16 core CPU nodes connected by basic low bandwidth interconnect. Moreover, the training time is at par with some of the best hardware accelerators.

Motivation & Objective

• To enable training large neural networks on underutilized, low-bandwidth CPU clusters that are commonly idle in cloud environments.
• To overcome the communication bottleneck in distributed model parallel training by reducing data movement through adaptive sparsity.
• To design a scalable, efficient framework that works on small, few-core CPUs with limited memory and low inter-node bandwidth.
• To outperform state-of-the-art distributed training frameworks like Horovod in low-resource, low-bandwidth settings.
• To demonstrate that CPU-based training can match or exceed the performance of high-end GPU accelerators in per-epoch training speed.

Proposed method

• D-SLIDE uses model parallelism to shard neural network layers across multiple CPU nodes, distributing both model weights and hash tables used for adaptive sparsity.
• It leverages the SLIDE algorithm’s ability to achieve over 95% sparsity per training step, drastically reducing the volume of gradients and parameters transmitted between nodes.
• Communication is optimized via MPI-based message passing, with only sparse updates (99% compressed) sent between nodes, minimizing bandwidth usage.
• The framework ensures load balancing by distributing hash tables and model parameters across nodes, enabling efficient computation and reduced contention.
• Sparsity is dynamically determined per input using a learnable hash function, maintaining high training efficiency even on CPUs with limited parallelism.
• The implementation is built on MPI and supports integration with common deep learning frameworks like PyTorch and TensorFlow.

Experimental results

Research questions

• RQ1 Can large neural networks be trained efficiently on low-bandwidth, low-resource CPU clusters that are typically underutilized in cloud environments?
• RQ2 To what extent can adaptive sparsity reduce communication overhead in distributed model-parallel training without sacrificing model accuracy?
• RQ3 How does D-SLIDE compare to Horovod in terms of training speed and scalability under low-bandwidth (≤1 Gbps) and low-core (4–16) conditions?
• RQ4 Can CPU-based training achieve per-epoch training times competitive with high-end GPU accelerators like A100 and V100?
• RQ5 Is it feasible to train near-billion-parameter models on 8–16 GB memory nodes using only model parallelism and sparsity?

Key findings

• On a 2-node cluster with 16 cores and 1 Gbps bandwidth, D-SLIDE trained a large model 80x faster than Horovod, which failed to scale beyond a single node due to communication bottlenecks.
• D-SLIDE achieved a per-epoch training time of 5,417 seconds on 8 nodes with 4 cores each, while Horovod required over 280,000 seconds on the same setup.
• On the Text8 dataset, D-SLIDE on 2 nodes was 6.7x faster than a single V100 GPU and 2.5x faster than a single A100 GPU per epoch, achieving comparable accuracy.
• On the Amazon670K dataset, D-SLIDE on 2 nodes was 3.0x faster than 2 V100 GPUs and 2.1x faster than a single A100 GPU per epoch, with test accuracy close to TensorFlow baselines.
• With 99% communication compression, D-SLIDE maintained strong scalability across 8 nodes, demonstrating robustness in low-bandwidth environments.
• The framework successfully trained an 800M-parameter model on 4–16 core CPUs with only 8–16 GB of memory per node, proving feasibility on minimal hardware.