[Paper Review] Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

Shaohuai Shi, Qiang Wang+1 more|arXiv (Cornell University)|Nov 16, 2017

Advanced Neural Network Applications31 references16 citations

TL;DR

This paper evaluates the performance of four leading distributed deep learning frameworks—Caffe-MPI, CNTK, MXNet, and TensorFlow—on a 4-node GPU cluster with Tesla P40 GPUs, modeling and benchmarking synchronous SGD across single-GPU, multi-GPU, and multi-node configurations. It identifies communication overhead, cuDNN tuning, and gradient aggregation hiding as key performance bottlenecks, with scaling efficiencies ranging from 35.6% to 77.7% due to network latency and suboptimal inter-node communication patterns.

ABSTRACT

Deep learning frameworks have been widely deployed on GPU servers for deep learning applications in both academia and industry. In training deep neural networks (DNNs), there are many standard processes or algorithms, such as convolution and stochastic gradient descent (SGD), but the running performance of different frameworks might be different even running the same deep model on the same GPU hardware. In this study, we evaluate the running performance of four state-of-the-art distributed deep learning frameworks (i.e., Caffe-MPI, CNTK, MXNet, and TensorFlow) over single-GPU, multi-GPU, and multi-node environments. We first build performance models of standard processes in training DNNs with SGD, and then we benchmark the running performance of these frameworks with three popular convolutional neural networks (i.e., AlexNet, GoogleNet and ResNet-50), after that, we analyze what factors that result in the performance gap among these four frameworks. Through both analytical and experimental analysis, we identify bottlenecks and overheads which could be further optimized. The main contribution is that the proposed performance models and the analysis provide further optimization directions in both algorithmic design and system configuration.

Motivation & Objective

• To analyze and model the performance of distributed deep learning frameworks under varying hardware configurations (single-GPU, multi-GPU, multi-node).
• To identify performance gaps among state-of-the-art frameworks (Caffe-MPI, CNTK, MXNet, TensorFlow) when training standard CNNs on identical hardware.
• To investigate the impact of data I/O, cuDNN tuning, and inter-node communication on training throughput and scaling efficiency.
• To provide performance models and actionable insights for optimizing framework design and system configuration in distributed DNN training.

Proposed method

• Developed analytical performance models for key components of synchronous SGD, including gradient computation, communication, and aggregation.
• Benchmarked four frameworks using three standard CNNs—AlexNet, GoogleNet, and ResNet-50—across single-GPU, multi-GPU, and multi-node environments.
• Measured and analyzed communication overhead using NCCL2.0 (CNTK), TCP sockets (MXNet), and gRPC (TensorFlow) for all-reduce operations.
• Quantified scaling efficiency by comparing actual training time against theoretical speedup, using metrics like $E_{allreduce}$ and $t_{comm}$.
• Evaluated the effectiveness of gradient communication hiding by measuring overlap between backward pass and all-reduce operations.
• Conducted experiments on a 4-node cluster with 56Gbps InfiniBand and 4x Tesla P40 GPUs per node, using controlled mini-batch sizes and input layouts.

Experimental results

Research questions

• RQ1 How do the four frameworks (Caffe-MPI, CNTK, MXNet, TensorFlow) compare in training throughput and scaling efficiency across single-GPU, multi-GPU, and multi-node setups?
• RQ2 What factors contribute to performance differences among frameworks despite using the same hardware and DNN models?
• RQ3 To what extent can gradient communication overhead be hidden, and how does this affect scaling performance in multi-GPU and multi-node environments?
• RQ4 How do cuDNN autotuning and input data layout (e.g., NCWH vs. NWHC) impact performance in forward and backward passes?
• RQ5 What is the role of network bandwidth and latency (e.g., 56Gbps InfiniBand) in limiting scalability across multiple machines?

Key findings

• For shallow networks like AlexNet, large mini-batch sizes can make data loading a bottleneck even with fast GPUs, highlighting the need for efficient data pre-processing.
• CNTK and MXNet achieve better performance through cuDNN autotuning and optimal input layout (NCWH), while TensorFlow and Caffe-MPI show lower efficiency due to suboptimal configuration.
• MXNet and TensorFlow hide gradient communication overhead by overlapping it with backward pass computation, resulting in better scaling; CNTK does not, leading to lower efficiency.
• Scaling efficiency drops significantly across four machines due to inter-node communication latency; CNTK achieves 55%, 67.5%, and 77.7% efficiency on AlexNet, GoogleNet, and ResNet-50 respectively.
• MXNet suffers from high communication overhead (0.4505s vs. 0.1819s backward pass in GoogleNet on four machines), resulting in only 35.6% scaling efficiency.
• TensorFlow achieves 50.6%, 75.6%, and 52.2% scaling efficiency on AlexNet, GoogleNet, and ResNet-50, respectively, due to high gRPC latency and suboptimal RDMA usage.