[Paper Review] Stochastic-Sign SGD for Federated Learning with Theoretical Guarantees

Richeng Jin, Yufan Huang+3 more|arXiv (Cornell University)|Feb 25, 2020

Privacy-Preserving Technologies in Data44 references43 citations

TL;DR

Presents Stochastic-Sign SGD with two compressors (sto-sign and dp-sign) to achieve convergence under data heterogeneity, plus Byzantine resilience and differential privacy in Federated Learning, with error-feedback variants and experiments on MNIST/CIFAR-10.

ABSTRACT

Federated learning (FL) has emerged as a prominent distributed learning paradigm. FL entails some pressing needs for developing novel parameter estimation approaches with theoretical guarantees of convergence, which are also communication efficient, differentially private and Byzantine resilient in the heterogeneous data distribution settings. Quantization-based SGD solvers have been widely adopted in FL and the recently proposed SIGNSGD with majority vote shows a promising direction. However, no existing methods enjoy all the aforementioned properties. In this paper, we propose an intuitively-simple yet theoretically-sound method based on SIGNSGD to bridge the gap. We present Stochastic-Sign SGD which utilizes novel stochastic-sign based gradient compressors enabling the aforementioned properties in a unified framework. We also present an error-feedback variant of the proposed Stochastic-Sign SGD which further improves the learning performance in FL. We test the proposed method with extensive experiments using deep neural networks on the MNIST dataset and the CIFAR-10 dataset. The experimental results corroborate the effectiveness of the proposed method.

Motivation & Objective

• Motivate Federated Learning’s challenges: communication efficiency, data heterogeneity, privacy, and Byzantine failures.
• Introduce stochastic-sign based gradient compressors to ensure convergence under heterogeneous data distributions.
• Provide privacy guarantees via a differentially private compressor and analyze convergence with error-feedback.
• Quantify Byzantine resilience and propose enhancements (weighted vote, Top-k sparsification) to improve robustness.
• Validate the approach with experiments on MNIST and CIFAR-10 using neural networks.

Proposed method

• Define sto-sign as a two-level stochastic quantization that transmits signs of quantized gradient results.
• Define dp-sign as a differentially private version of sto-sign to achieve (epsilon, delta)-DP.
• Propose DP-SIGN SGD and Sto-SIGN SGD with convergence analysis under heterogeneous data distributions.
• Incorporate an error-feedback mechanism to compensate for majority-vote induced errors and prove convergence for the variant.
• Establish Byzantine resilience bounds and discuss reputation-based weighting and robustness enhancements.
• Extend the framework to handle robustness against attackers and analyze performance under data heterogeneity.

Experimental results

Research questions

• RQ1 How can sign-based gradient methods converge when data is heterogeneously distributed across workers?
• RQ2 What stochastic gradient compression schemes can ensure convergence and maintain communication efficiency in FL?
• RQ3 How can differential privacy be integrated into sign-based gradient compression without harming performance?
• RQ4 What is the Byzantine resilience level of Stochastic-Sign SGD and how can it be enhanced?

Key findings

• Sto-SIGN SGD converges to a neighborhood of the local optimum under heterogeneous data, with gap diminishing as the number of workers M grows.
• Averaged wrong-sign probability can be bounded using the stochastic compressor, enabling convergence where SIGN SGD fails.
• DP-SIGN SGD provides differential privacy guarantees while maintaining communication efficiency and accuracy.
• Error-Feedback Stochastic-Sign SGD further improves learning performance with proven convergence.
• The paper derives Byzantine resilience bounds that depend on data heterogeneity and proposes weighted vote variants to improve robustness.
• Experimental results on MNIST and CIFAR-10 with deep nets validate effectiveness and robustness under communication constraints.