[Paper Review] Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD

Sanghamitra Dutta, Gauri Joshi+3 more|arXiv (Cornell University)|Mar 3, 2018

Distributed and Parallel Computing SystemsComputer Science94 citations

TL;DR

The paper analyzes the error-runtime trade-offs of synchronous and asynchronous distributed SGD variants under random straggler delays, proposing a learning-rate schedule to compensate for gradient staleness and assessing runtimes without bounded-delay assumptions.

ABSTRACT

Distributed Stochastic Gradient Descent (SGD) when run in a synchronous manner, suffers from delays in waiting for the slowest learners (stragglers). Asynchronous methods can alleviate stragglers, but cause gradient staleness that can adversely affect convergence. In this work we present a novel theoretical characterization of the speed-up offered by asynchronous methods by analyzing the trade-off between the error in the trained model and the actual training runtime (wallclock time). The novelty in our work is that our runtime analysis considers random straggler delays, which helps us design and compare distributed SGD algorithms that strike a balance between stragglers and staleness. We also present a new convergence analysis of asynchronous SGD variants without bounded or exponential delay assumptions, and a novel learning rate schedule to compensate for gradient staleness.

Motivation & Objective

• Motivate the study by addressing stragglers and gradient staleness in distributed SGD.
• Model the wallclock runtime per iteration as a random variable with general distribution.
• Provide convergence analyses for asynchronous SGD variants without bounded delay assumptions.
• Propose a novel learning rate schedule to compensate for gradient staleness and improve stability.

Proposed method

• Define a central parameter server model with P learners and i.i.d. per-mini-batch runtimes X_i.
• Characterize and compare SGD variants: K-sync, K-batch-sync, K-async, and K-batch-async.
• Analyze expected runtime per iteration using order statistics and renewal theory.
• Provide a convergence analysis for Async and K-async SGD under relaxed assumptions (no bounded delays, no exponential runtimes).
• Introduce a variable learning rate schedule to adapt to gradient staleness and prove its stability properties.

Experimental results

Research questions

• RQ1 How does the expected wallclock runtime per iteration compare across synchronous and asynchronous SGD variants under general, stochastic runtimes?
• RQ2 What is the impact of gradient staleness on convergence for Async and K-async SGD, and how can learning rate schedules mitigate it?
• RQ3 Under relaxed assumptions (no bounded delays, general distributions), what are the convergence guarantees for asynchronous SGD variants?
• RQ4 Can K-batch-async offer better error-runtime trade-offs than K-async or K-sync in practice and theory?
• RQ5 What regimes cause asynchronous methods to outperform synchronous methods in terms of wallclock efficiency?

Key findings

• Asynchronous SGD can yield faster wallclock progress than synchronous SGD, with speed-up characterized by P times a factor depending on the delay distribution.
• K-batch-async and K-batch-sync variants improve runtime efficiency by reducing idle time without sacrificing convergence behavior significantly.
• For exponential runtimes, the runtime speed-up between sync and async scales roughly as P log P, indicating substantial advantages with many learners.
• A new convergence bound for K-async SGD under general staleness bounds shows its error decays at a rate dependent on (1 - eta c (1 - gamma + p0/2)).
• A variable learning rate schedule that scales with gradient staleness stabilizes asynchronous SGD and can outperform fixed-rate approaches in practice.