[Paper Review] Error Feedback Fixes SignSGD and other Gradient Compression Schemes

Sai Praneeth Karimireddy, Quentin Rebjock+2 more|arXiv (Cornell University)|Jan 28, 2019

Stochastic Gradient Optimization Techniques153 citations

TL;DR

The paper shows that biased gradient compression methods (like SignSGD) may fail to converge or generalize poorly, and introduces EF-SGD with error-feedback to recover SGD’s convergence rate and improve generalization, even with compression.

ABSTRACT

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD does not converge to the optimum. Further, even when it does converge, signSGD may generalize poorly when compared with SGD. These issues arise because of the biased nature of the sign compression operator. We then show that using error-feedback, i.e. incorporating the error made by the compression operator into the next step, overcomes these issues. We prove that our algorithm EF-SGD with arbitrary compression operator achieves the same rate of convergence as SGD without any additional assumptions. Thus EF-SGD achieves gradient compression for free. Our experiments thoroughly substantiate the theory and show that error-feedback improves both convergence and generalization. Code can be found at \url{https://github.com/epfml/error-feedback-SGD}.

Motivation & Objective

• Demonstrate limitations of sign-based gradient compression such as SIGNSGD, including potential non-convergence and poor generalization.
• Propose and analyze an error-feedback mechanism (EF-SGD) that enables convergence and preserves SGD rates under compression.
• Provide theoretical guarantees for both non-convex and non-smooth settings.
• Empirically validate that EF-SG​D matches SGD performance with substantial communication savings.

Proposed method

• Introduce EF-SIGNSGD, a SignSGD variant with error-feedback that incorporates accumulated compression error into the next step.
• Model the compressor as a δ-approximate operator and prove EF-SGD converges at SGD-like rates under standard smoothness and variance assumptions.
• Show that error feedback keeps the iterates in the span of gradients and mitigates bias from compression.
• Extend the analysis to non-convex and non-smooth settings with explicit convergence bounds.
• Empirically validate on CIFAR-10/100 using VGG and ResNet, comparing against SGD, SIGNSGD, and SIGNSGDM.

Experimental results

Research questions

• RQ1 Can biased gradient compression schemes like SignSGD converge and generalize as well as SGD, or do they inherently fail under standard assumptions?
• RQ2 Does adding error-feedback to compressed gradients restore convergence and improve generalization to match SGD?
• RQ3 What are the theoretical convergence rates of EF-SGD under smooth and non-smooth settings with compression?
• RQ4 How does EF-SIGNSGD perform empirically in deep networks compared to SGD and other sign-based methods, across batch sizes?

Key findings

• SignSGD can fail to converge and generalization can be worse than SGD under biased gradient compression.
• Incorporating error-feedback (EF-SGD) yields convergence at rates matching SGD for non-convex smooth objectives, effectively giving compression for free.
• EF-SIGNSGD preserves iterates within the span of past gradients, which improves generalization to near SGD levels in over-parameterized settings.
• Empirical results on CIFAR-10/100 with ResNet/VGG show EF-SIGNSGD outperforms SIGNSGD and SIGNSGDM and closely matches SGD, while reducing communication by about 64x.
• Theoretical results extend to non-smooth convex cases, with compression error affecting the leading term in convergence rates.
• EF-SIGNSGD generalization gap on CIFAR-100 is substantially smaller than SIGNSGD/ALL alternatives, especially at smaller batch sizes.