[Paper Review] Dynamic Stale Synchronous Parallel Distributed Training for Deep Learning

Xing Zhao, Aijun An+2 more|arXiv (Cornell University)|Aug 16, 2019

Advanced Neural Network Applications37 references4 citations

TL;DR

This paper proposes Dynamic Stale Synchronous Parallel (DSSP), a novel distributed training paradigm for deep learning that dynamically adjusts the staleness threshold at runtime based on workers' processing times, eliminating the need for manual hyperparameter tuning. DSSP improves convergence speed and accuracy—especially in heterogeneous GPU environments—by reducing synchronization wait times while maintaining convergence guarantees, outperforming SSP, BSP, and ASP in both stability and performance.

ABSTRACT

Deep learning is a popular machine learning technique and has been applied to many real-world problems. However, training a deep neural network is very time-consuming, especially on big data. It has become difficult for a single machine to train a large model over large datasets. A popular solution is to distribute and parallelize the training process across multiple machines using the parameter server framework. In this paper, we present a distributed paradigm on the parameter server framework called Dynamic Stale Synchronous Parallel (DSSP) which improves the state-of-the-art Stale Synchronous Parallel (SSP) paradigm by dynamically determining the staleness threshold at the run time. Conventionally to run distributed training in SSP, the user needs to specify a particular staleness threshold as a hyper-parameter. However, a user does not usually know how to set the threshold and thus often finds a threshold value through trial and error, which is time-consuming. Based on workers' recent processing time, our approach DSSP adaptively adjusts the threshold per iteration at running time to reduce the waiting time of faster workers for synchronization of the globally shared parameters, and consequently increases the frequency of parameters updates (increases iteration throughput), which speedups the convergence rate. We compare DSSP with other paradigms such as Bulk Synchronous Parallel (BSP), Asynchronous Parallel (ASP), and SSP by running deep neural networks (DNN) models over GPU clusters in both homogeneous and heterogeneous environments. The results show that in a heterogeneous environment where the cluster consists of mixed models of GPUs, DSSP converges to a higher accuracy much earlier than SSP and BSP and performs similarly to ASP. In a homogeneous distributed cluster, DSSP has more stable and slightly better performance than SSP and ASP, and converges much faster than BSP.

Motivation & Objective

• To address the challenge of manually selecting an optimal staleness threshold in Stale Synchronous Parallel (SSP) training, which is time-consuming and often suboptimal.
• To reduce synchronization wait times for faster workers in distributed deep learning by adapting the staleness threshold at runtime based on real-time worker performance.
• To improve training convergence speed and accuracy in heterogeneous environments where GPU processing capabilities vary significantly.
• To maintain theoretical convergence guarantees while enhancing practical flexibility and stability compared to existing paradigms like BSP, ASP, and SSP.
• To provide a lightweight, adaptive mechanism that adjusts thresholds per iteration without requiring prior knowledge of system heterogeneity.

Proposed method

• DSSP dynamically determines the staleness threshold for each worker at every iteration using real-time processing time measurements from workers.
• The algorithm computes a local staleness threshold $ s_L $ per worker based on recent processing times, enabling faster workers to proceed without waiting for slower ones beyond a dynamically adjusted limit.
• A global threshold $ r $ is used to bound the maximum allowable staleness across all workers, ensuring convergence stability.
• The method integrates with the parameter server framework, where workers push gradients and pull updated global weights, with synchronization controlled by the dynamic threshold.
• The approach uses a lightweight runtime adaptation mechanism that avoids expensive retraining or hyperparameter search.
• Theoretical analysis shows DSSP inherits SSP’s regret bound, ensuring convergence as long as the threshold range remains constant.

Experimental results

Research questions

• RQ1 How can the staleness threshold in SSP be automatically determined at runtime to reduce manual tuning overhead?
• RQ2 Can dynamic threshold adjustment improve convergence speed and accuracy in heterogeneous GPU clusters?
• RQ3 How does DSSP compare to BSP, ASP, and SSP in terms of training speed, accuracy, and stability across homogeneous and heterogeneous environments?
• RQ4 What is the impact of fully connected layers on the convergence trends of different distributed paradigms?
• RQ5 Can DSSP maintain strong performance under fluctuating network conditions or unstable worker speeds?

Key findings

• In a heterogeneous GPU cluster with GTX1060 and GTX1080 Ti, DSSP converged to 0.68 test accuracy in 3046.3 seconds, significantly faster than SSP with $ s=15 $, which took 7255.6 seconds.
• DSSP achieved higher test accuracy than SSP and BSP in the heterogeneous environment, while ASP, though fastest, converged to a lower accuracy due to excessive staleness.
• On a homogeneous cluster, DSSP showed more stable and slightly better performance than SSP and ASP, and converged much faster than BSP.
• DSSP reached 0.67 accuracy in 3016.4 seconds, matching ASP’s speed but with better accuracy and convergence stability.
• For ResNet-110 on CIFAR-100, DSSP outperformed SSP with $ s=3 $, achieving higher accuracy faster, even when SSP was tuned with optimal thresholds.
• The results indicate that DSSP maintains consistent performance across varying hardware configurations, unlike ASP, which suffers from high staleness in imbalanced environments.