[Paper Review] Performance Modeling of Distributed Deep Neural Networks

Sayed Hadi Hashemi, Shadi A. Noghabi+2 more|arXiv (Cornell University)|Dec 2, 2016

Advanced Memory and Neural Computing5 references3 citations

TL;DR

This paper proposes a performance modeling and profiling study of Microsoft CNTK, a distributed deep neural network framework, revealing that poor I/O utilization and a single-server bottleneck severely limit scalability. Despite theoretical linear scaling, CNTK degrades beyond 4 nodes due to sequential disk I/O and server saturation, with computation time increasing after 16 workers, highlighting critical inefficiencies in distributed DNN systems.

ABSTRACT

During the past decade, machine learning has become extremely popular and can be found in many aspects of our every day life. Nowayadays with explosion of data while rapid growth of computation capacity, Distributed Deep Neural Networks (DDNNs) which can improve their performance linearly with more computation resources, have become hot and trending. However, there has not been an in depth study of the performance of these systems, and how well they scale. In this paper we analyze CNTK, one of the most commonly used DDNNs, by first building a performance model and then evaluating the system two settings: a small cluster with all nodes in a single rack connected to a top of rack switch, and in large scale using Blue Waters with arbitary placement of nodes. Our main focus was the scalability of the system with respect to adding more nodes. Based on our results, this system has an excessive initialization overhead because of poor I/O utilization which dominates the whole execution time. Because of this, the system does not scale beyond a few nodes (4 in Blue Waters). Additionally, due to a single server-multiple worker design the server becomes a bottleneck after 16 nodes limiting the scalability of the CNTK.

Motivation & Objective

• To develop a performance model for Asynchronous Stochastic Gradient Descent (ASGD) in distributed DNNs.
• To evaluate CNTK's actual performance against the theoretical model on both small and large-scale clusters.
• To identify and diagnose scalability bottlenecks in CNTK through detailed profiling and tracing.
• To propose improvements targeting I/O inefficiency and server overload in distributed training frameworks.

Proposed method

• Built a performance model accounting for CPU, memory, disk I/O, and network communication in ASGD-based DNN training.
• Conducted scalability experiments on an 8-node cluster and the Blue Waters supercomputer with varying worker counts.
• Used the TAU performance tool to profile and trace execution phases, isolating I/O and server-related overhead.
• Updated the initial performance model to reflect sequential disk access and server saturation, introducing new equations for T_new_disk and T_new_total.
• Analyzed phase-wise execution (initialization and computation) to isolate scalability trends and bottlenecks.
• Proposed future improvements focusing on parallel I/O and distributed parameter server architectures to alleviate server bottlenecks.

Experimental results

Research questions

• RQ1 Why does CNTK fail to scale beyond 4–16 nodes despite its distributed design?
• RQ2 To what extent does I/O performance, particularly sequential disk access, dominate execution time in initialization?
• RQ3 How does the single-server architecture become a bottleneck as the number of workers increases?
• RQ4 How well does the initial performance model predict real-world execution compared to the updated model with I/O and server constraints?
• RQ5 What are the key performance gaps between theoretical scalability and actual system behavior in distributed DNN training?

Key findings

• Initialization time increases linearly with the number of nodes due to sequential disk I/O, making it the dominant performance overhead.
• Computation time decreases linearly up to 16 nodes but increases beyond that point due to server saturation.
• The system does not scale effectively beyond 4 nodes on Blue Waters and beyond 16 nodes due to server bottleneck.
• The updated performance model, which accounts for sequential I/O and server limits, closely matches experimental results.
• Even with perfect parallelization of initialization, the system cannot scale beyond 16 nodes due to server constraints.
• The performance gap between theoretical and actual scalability is primarily due to I/O inefficiency and monolithic server architecture.