[Paper Review] Priority-based Parameter Propagation for Distributed DNN Training

Anand Jayarajan, Jinliang Wei+3 more|arXiv (Cornell University)|May 10, 2019

Advanced Neural Network Applications22 references42 citations

TL;DR

P3 introduces parameter slicing and priority-based updates to overlap communication with computation, improving data-parallel DNN training under limited bandwidth without affecting convergence.

ABSTRACT

Data parallel training is widely used for scaling distributed deep neural network (DNN) training. However, the performance benefits are often limited by the communication-heavy parameter synchronization step. In this paper, we take advantage of the domain specific knowledge of DNN training and overlap parameter synchronization with computation in order to improve the training performance. We make two key observations: (1) the optimal data representation granularity for the communication may differ from that used by the underlying DNN model implementation and (2) different parameters can afford different synchronization delays. Based on these observations, we propose a new synchronization mechanism called Priority-based Parameter Propagation (P3). P3 synchronizes parameters at a finer granularity and schedules data transmission in such a way that the training process incurs minimal communication delay. We show that P3 can improve the training throughput of ResNet-50, Sockeye and VGG-19 by as much as 25%, 38% and 66% respectively on clusters with realistic network bandwidth

Motivation & Objective

• Motivate the need to overcome communication bottlenecks in data-parallel DNN training with synchronous SGD.
• Leverage domain-specific knowledge of DNN training to overlap parameter synchronization with computation.
• Show that finer-grained parameter synchronization can outperform layer-level methods under bandwidth constraints.
• Ensure the method maintains convergence while reducing communication overhead.

Proposed method

• Propose Parameter Slicing: split layers into smaller parameter slices and synchronize them independently.
• Apply Priority-based Update: assign priorities to slices based on when they are needed in the next iteration and schedule high-priority slices first.
• Implement within MXNet KVStore by introducing P3 Worker and P3 Server with a producer-consumer priority queue for slices.
• Use a round-robin assignment of slices to P3 Servers and remove explicit per-slice update notifications to enable bidirectional bandwidth utilization.
• Maintain full-gradient synchronization to avoid affecting model convergence.

Experimental results

Research questions

• RQ1 Can finer-grained parameter synchronization reduce communication delay compared to layer-level synchronization under limited bandwidth?
• RQ2 Does incorporating layer consumption timing (forward pass requirements) into scheduling further improve overlap of communication and computation?
• RQ3 Does P3 preserve SGD convergence while improving training throughput across models with varying layer granularities?

Key findings

• P3 improves training throughput for ResNet-50, Sockeye, and VGG-19 by up to 25%, 38%, and 66% respectively under realistic network bandwidths.
• P3 maintains linear throughput down to lower bandwidths than the baseline in several models, showing better resilience to bandwidth limits.
• Parameter slicing reduces peak bandwidth usage and enables more effective bidirectional bandwidth utilization.
• Compared with compression-based methods like DGC, P3 achieves better accuracy and does not modify the SGD convergence behavior.
• P3 yields large throughput gains for models with very large layers (e.g., VGG-19) and substantial savings even when some models see smaller gains.