[Paper Review] SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

Beidi Chen, Tharun Medini+4 more|arXiv (Cornell University)|Mar 7, 2019

Advanced Neural Network Applications43 references44 citations

TL;DR

SLIDE presents a CPU-based sparse, locality-sensitive hashing approach that enables multi-core parallelism to outperform GPU-accelerated TensorFlow on large fully connected networks, achieving up to 3.5x–10x wall-clock speedups at similar accuracy.

ABSTRACT

Deep Learning (DL) algorithms are the central focus of modern machine learning systems. As data volumes keep growing, it has become customary to train large neural networks with hundreds of millions of parameters to maintain enough capacity to memorize these volumes and obtain state-of-the-art accuracy. To get around the costly computations associated with large models and data, the community is increasingly investing in specialized hardware for model training. However, specialized hardware is expensive and hard to generalize to a multitude of tasks. The progress on the algorithmic front has failed to demonstrate a direct advantage over powerful hardware such as NVIDIA-V100 GPUs. This paper provides an exception. We propose SLIDE (Sub-LInear Deep learning Engine) that uniquely blends smart randomized algorithms, with multi-core parallelism and workload optimization. Using just a CPU, SLIDE drastically reduces the computations during both training and inference outperforming an optimized implementation of Tensorflow (TF) on the best available GPU. Our evaluations on industry-scale recommendation datasets, with large fully connected architectures, show that training with SLIDE on a 44 core CPU is more than 3.5 times (1 hour vs. 3.5 hours) faster than the same network trained using TF on Tesla V100 at any given accuracy level. On the same CPU hardware, SLIDE is over 10x faster than TF. We provide codes and scripts for reproducibility.

Motivation & Objective

• Motivate the search for algorithmic alternatives to hardware acceleration in large-scale deep learning.
• Propose a practical, CPU-based system (SLIDE) that exploits adaptive sparsity to reduce computation.
• Show that smart algorithms can outperform GPU-accelerated baselines on industry-scale datasets.
• Provide reproducible code and benchmarks to validate the approach.
• Analyze performance characteristics and bottlenecks of the proposed system.

Proposed method

• Deploy Locality Sensitive Hashing (LSH) to sparsify neuron activations and enable sub-linear candidate selection.
• Use K LSH hashes per layer to generate sparse neuron subsets for forward passes.
• Perform sparse backpropagation by updating only active connections and employing asynchronous SGD (HOGWILD-style).
• Leverage multi-core OpenMP parallelism with batch-level independence to achieve near-linear scaling.
• Incorporate memory and cache-aware optimizations (e.g., HugePages, SIMD) to speed up CPU execution.
• Compare SLIDE against TF-GPU and TF-CPU on large fully connected networks with datasets Delicious-200K and Amazon-670K.

Experimental results

Research questions

• RQ1 Can algorithmic sparsification via LSH outperform hardware-accelerated training for large neural networks on CPU cores?
• RQ2 What is the impact of adaptive neuron sampling on convergence and accuracy compared to full or sampled softmax baselines?
• RQ3 How does SLIDE scale with increasing CPU cores and dataset size in wall-clock time and core utilization?
• RQ4 What are the practical bottlenecks (memory, bandwidth) and how can they be mitigated in a CPU-based DL system?

Key findings

• SLIDE on a 44-core CPU outperforms TF-Tesla V100 GPU in wall-clock time at equivalent accuracy.
• On Delicious-200K, SLIDE is about 1.8x faster than TF-GPU; on Amazon-670K, about 2.7x faster.
• SLIDE is over 10x faster than TF-CPU, and converges with similar iteration-to-accuracy behavior.
• Memory-bound inefficiencies decrease as cores increase for SLIDE, while they rise for TF-CPU, enabling better CPU utilization.
• Adaptive sampling via LSH enables significant reduction in active neurons and updates, driving large speedups with minimal accuracy loss.
• SLIDE achieves near-peak core utilization (~80–85%) across 8–32 threads, outperforming TF-CPU in efficiency.