[Paper Review] Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD

Jianyu Wang, Gauri Joshi|arXiv (Cornell University)|Oct 18, 2018

Distributed and Parallel Computing SystemsComputer Science119 citations

TL;DR

The paper introduces AdaComm, an adaptive communication strategy for local-update SGD that starts with infrequent averaging and gradually increases communication to achieve fast error convergence with a low final error; experiments show up to 3x runtime speedups over fully synchronous SGD while attaining the same final training loss.

ABSTRACT

Large-scale machine learning training, in particular distributed stochastic gradient descent, needs to be robust to inherent system variability such as node straggling and random communication delays. This work considers a distributed training framework where each worker node is allowed to perform local model updates and the resulting models are averaged periodically. We analyze the true speed of error convergence with respect to wall-clock time (instead of the number of iterations), and analyze how it is affected by the frequency of averaging. The main contribution is the design of AdaComm, an adaptive communication strategy that starts with infrequent averaging to save communication delay and improve convergence speed, and then increases the communication frequency in order to achieve a low error floor. Rigorous experiments on training deep neural networks show that AdaComm can take $3 \ imes$ less time than fully synchronous SGD, and still reach the same final training loss.

Motivation & Objective

• Motivate and analyze the error versus wall-clock time convergence of distributed SGD with local updates and periodic averaging.
• Quantify how averaging frequency (tau) affects runtime per iteration and error floor.
• Develop an adaptive communication scheme (AdaComm) that optimizes the trade-off for real-world training.
• Provide theoretical convergence insights for PASGD with variable tau and learning rate.
• Demonstrate practical benefits of AdaComm on deep CNNs under realistic system variability.

Proposed method

• Model the runtime per iteration of PASGD under random local compute times and a random communication delay.
• Derive the error-runtime bound for PASGD as a function of tau, leading to an expression for the optimal tau.
• Propose AdaComm which partitions training into time intervals and selects tau per interval to minimize bound-based error.
• Provide practical tau update rules that do not require unknown constants (use a loss-ratio heuristic).
• Extend the analysis to include decaying tau and adaptive learning rate scenarios.
• Validate AdaComm with experiments on VGG-16 and ResNet-50 using CIFAR-10/100 datasets.

Experimental results

Research questions

• RQ1 How does the local-update frequency tau influence true convergence speed in wall-clock time for PASGD?
• RQ2 Can adaptive communication that varies tau over time outperform fixed-tau schemes in terms of error-runtime trade-off?
• RQ3 What is the optimal tau as a function of time, data, and system delays to minimize the bound on gradient norm?
• RQ4 How can practical heuristics implement AdaComm without access to unknown constants like Lipschitz constants or gradient variance bounds?
• RQ5 Do adaptive communication strategies generalize to different network architectures and learning-rate schedules?

Key findings

• AdaComm achieves faster wall-clock convergence by starting with larger tau and decreasing it as training progresses.
• Theoretical analysis shows an error-runtime trade-off where larger tau reduces per-iteration runtime but can increase the error floor; AdaComm mitigates this by adapting tau over time.
• Experiments on VGG-16 and ResNet-50 show AdaComm yields up to ~3x runtime speed-up over fully synchronous SGD while reaching the same final training loss (and better test accuracy in some settings).
• A closed-form expression identifies an optimal tau* under simplifying constants, guiding practical adaptation of communication frequency.
• AdaComm can be combined with learning-rate schedules and remains applicable to related communication-efficient SGD frameworks.