[Paper Review] Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

Tal Ben‐Nun, Torsten Hoefler|arXiv (Cornell University)|Feb 26, 2018

Stochastic Gradient Optimization Techniques253 references212 citations

TL;DR

A comprehensive survey analyzing concurrency in deep learning from single-operator to distributed-scale training, with models and implications for parallelism strategies.

ABSTRACT

Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.

Motivation & Objective

• Define terminology and foundational algorithms for parallel and distributed deep learning.
• Analyze concurrency across DNN operators, network architectures, and training/inference workflows.
• Review parallel computer architectures, communication schemes, and system implementations relevant to distributed DL.
• Model concurrency using the Work-Depth framework and identify trends driving parallelization strategies.

Proposed method

• Survey and classification of DNN operators and their computation models.
• Convolutional, pooling, and normalization operator formulations with associated tensor data flows.
• Discussion of stochastic optimization and weight-update rules, including SGD and minibatch SGD with backpropagation.
• Application of the Work-Depth model to characterize parallelism and derive bounds for DAG-based computations.
• Analysis of single-machine versus multi-machine parallelism, including MPI and RDMA-based communication.

Experimental results

Research questions

• RQ1 How do DNN operators expose concurrency and what are the implications for parallelism?
• RQ2 What are the tradeoffs between concurrency, accuracy, and hardware utilization in minibatch SGD?
• RQ3 How do distributed architectures and communication strategies affect scalable training and inference of DNNs?
• RQ4 What role do parallel programming models and libraries (e.g., MPI, CUDA, Spark) play in distributed deep learning?
• RQ5 What future directions emerge for achieving higher parallelism in deep learning workloads?

Key findings

• GPU-accelerated nodes dominate DL research and distributed-memory systems are increasingly essential for large-scale training.
• Allreduce and other collective communication patterns are central bottlenecks in distributed DL and benefit from optimized HPC techniques.
• Minibatch size critically balances statistical generalization and hardware utilization, with theory and empirical evidence guiding warmup, learning-rate schedules, and variance control.
• Convolutional, pooling, and normalization operators are the primary computational kernels driving DL workloads and their parallelization strategies.
• DNN training and inference can be mapped to DAGs with work and depth characterizations, enabling parallelism analysis via the Work-Depth model.