[Paper Review] A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks

Daniel Nichols, Siddharth Singh+2 more|arXiv (Cornell University)|Nov 9, 2021

Advanced Neural Network Applications4 citations

TL;DR

This paper surveys and empirically evaluates leading parallel deep learning frameworks, comparing data parallelism, intra-layer, and inter-layer parallelism across vision and language tasks. It finds data parallelism offers superior statistical efficiency and scalability, while ZeRO reduces memory by 42–66% without performance loss, and hybrid approaches (e.g., Megatron + data parallelism) are optimal for large models on high-GPU counts.

ABSTRACT

The field of deep learning has witnessed a remarkable shift towards extremely compute- and memory-intensive neural networks. These newer larger models have enabled researchers to advance state-of-the-art tools across a variety of fields. This phenomenon has spurred the development of algorithms for distributed training of neural networks over a larger number of hardware accelerators. In this paper, we discuss and compare current state-of-the-art frameworks for large scale distributed deep learning. First, we survey current practices in distributed learning and identify the different types of parallelism used. Then, we present empirical results comparing their performance on large image and language training tasks. Additionally, we address their statistical efficiency and memory consumption behavior. Based on our results, we discuss algorithmic and implementation portions of each framework which hinder performance.

Motivation & Objective

• To provide a comprehensive survey of current state-of-the-art distributed deep learning frameworks and their parallelization strategies.
• To empirically evaluate open-source frameworks on large-scale image and language training tasks to compare runtime, memory consumption, and statistical efficiency.
• To identify algorithmic and implementation bottlenecks that hinder performance in modern deep learning frameworks.
• To guide researchers and practitioners in selecting optimal frameworks based on model size, hardware constraints, and training efficiency goals.

Proposed method

• Classified parallelization strategies into three categories: data parallelism, intra-layer (model) parallelism, and inter-layer (pipelining) parallelism.
• Selected and benchmarked six open-source frameworks: DDP, PipeDream, ZeRO, Megatron, TorchGPipe, and LBANN on two clusters with A100 and V100 GPUs.
• Measured epoch execution time, statistical efficiency (via validation accuracy/perplexity over time), and per-GPU memory usage across multiple GPU counts.
• Used standard benchmarks: VGG-16 on ImageNet and GPT2-medium on a language modeling dataset to evaluate scalability and convergence.
• Analyzed the impact of effective batch size increases in data parallelism and weight staleness in pipelined frameworks.
• Plotted validation metrics against training time to assess combined performance and statistical efficiency, avoiding reliance on epoch count alone.

Experimental results

Research questions

• RQ1 How do different parallelization strategies (data, intra-layer, inter-layer) compare in terms of training speed, memory usage, and statistical efficiency?
• RQ2 What is the impact of increasing GPU count on the performance and memory consumption of each framework?
• RQ3 How do weight staleness and gradient synchronization mechanisms affect convergence speed and statistical efficiency in pipelined frameworks?
• RQ4 To what extent do memory optimization techniques like ZeRO reduce GPU memory footprint without sacrificing training speed?
• RQ5 What is the optimal hybrid strategy (e.g., combining intra-layer and data parallelism) for training large models on high-GPU-count systems?

Key findings

• Data parallelism frameworks (DDP, ZeRO, LBANN) achieved the highest statistical efficiency, with validation curves nearly identical and convergence speeds superior to pipelined methods.
• PipeDream exhibited significantly slower convergence due to unmitigated weight staleness, even with its weight stashing technique, making it less effective for large-scale training.
• ZeRO reduced per-GPU memory usage by 42–66% compared to DDP, with memory savings increasing as GPU count grew, without sacrificing training speed.
• Megatron-intra showed better statistical efficiency than data parallel frameworks on the language task, but only at low GPU counts; performance degradation occurred at scale due to increasing effective batch size.
• The U-shaped memory curve in Megatron was caused by increasing activation memory at higher GPU counts, which eventually offset parameter memory savings from layer partitioning.
• Hybrid approaches combining intra-layer parallelism (e.g., Megatron) with data parallelism delivered optimal performance and memory efficiency for large models, especially on high-GPU systems.