[Paper Review] The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication

Sebastian U. Stich, Sai Praneeth Karimireddy|arXiv (Cornell University)|Sep 11, 2019

Stochastic Gradient Optimization TechniquesComputer Science65 references78 citations

TL;DR

The paper develops an error-feedback framework to analyze SGD with delayed and compressed gradients, showing that delay mainly affects a higher-order term and does not alter the optimal stochastic convergence rate under noise, and extends results to compressed gradients with error compensation and local SGD.

ABSTRACT

We analyze (stochastic) gradient descent (SGD) with delayed updates on smooth quasi-convex and non-convex functions and derive concise, non-asymptotic, convergence rates. We show that the rate of convergence in all cases consists of two terms: (i) a stochastic term which is not affected by the delay, and (ii) a higher order deterministic term which is only linearly slowed down by the delay. Thus, in the presence of noise, the effects of the delay become negligible after a few iterations and the algorithm converges at the same optimal rate as standard SGD. This result extends a line of research that showed similar results in the asymptotic regime or for strongly-convex quadratic functions only. We further show similar results for SGD with more intricate form of delayed gradients -- compressed gradients under error compensation and for local~SGD where multiple workers perform local steps before communicating with each other. In all of these settings, we improve upon the best known rates. These results show that SGD is robust to compressed and/or delayed stochastic gradient updates. This is in particular important for distributed parallel implementations, where asynchronous and communication efficient methods are the key to achieve linear speedups for optimization with multiple devices.

Motivation & Objective

• Motivate and analyze SGD with delayed/stale updates on smooth quasi-convex and non-convex functions.
• Derive non-asymptotic convergence rates that separate a noise-driven term from a delay-affected deterministic term.
• Extend the analysis to compressed gradients with error compensation and to local SGD.
• Provide tight iteration complexity results across quasi-convex and non-convex settings.
• Offer insights for distributed optimization where asynchronous updates and communication efficiency are key.

Proposed method

• Introduce an error-feedback (EC-SGD) framework with auxiliary sequences and a virtual iterate to capture delays and compression.
• Derive a one-step progress bound for the virtual iterates using perturbed iterate analysis and standard smoothness estimates.
• Decompose error into bias and noise components to allow larger step sizes and tighten delay-dependent terms.
• Provide non-asymptotic iteration complexity results for strongly quasi-convex, general quasi-convex, and non-convex cases.
• Extend the analysis to gradient compression with error compensation and to local SGD variants.
• Outline three-part proof strategy: one-step progress, final complexity via established techniques, and error bias/noise splitting.

Experimental results

Research questions

• RQ1 How do delayed gradient updates impact SGD convergence on quasi-convex and non-convex objectives?
• RQ2 Can the error-feedback framework yield non-asymptotic convergence rates that separate stochastic and delay terms?
• RQ3 Do compressed gradients with error compensation and local SGD admit similar favorable rates under this framework?
• RQ4 What are the iteration complexities for strongly quasi-convex, general quasi-convex, and non-convex settings under delays or compression?
• RQ5 How do these results compare to existing asymptotic or quadratic-function-specific analyses?

Key findings

• The convergence rate comprises a stochastic term unaffected by delay and a higher-order deterministic term slowed linearly by delay, making delay effects negligible under noise.
• The framework extends to compressed gradients with error compensation and to local SGD, improving upon prior rates.
• For strongly quasi-convex and general quasi-convex functions, the paper derives iteration complexities with explicit delay dependence, showing near-optimal scaling up to logarithmic factors.
• For general smooth non-convex functions, the method yields rates for converging to stationary points with explicit delay and noise terms.
• The analysis does not require bounded gradient assumptions in the non-convex case and improves over previous quadratic-dependence results on compression parameters.
• Follow-up work by others built on these techniques to further improve local SGD rates and establish lower bounds.