[Paper Review] Generalized Byzantine-tolerant SGD

Cong Xie, Oluwasanmi Koyejo+1 more|arXiv (Cornell University)|Feb 27, 2018

Distributed systems and fault toleranceComputer Science12 references176 citations

TL;DR

The paper introduces three robust, median-based aggregation rules for synchronous SGD under a generalized Byzantine failure model, proving dimensional Byzantine resilience and showing empirical robustness against diverse attacks.

ABSTRACT

We propose three new robust aggregation rules for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience properties of these aggregation rules. Empirical analysis shows that the proposed techniques outperform current approaches for realistic use cases and Byzantine attack scenarios.

Motivation & Objective

• Motivate failure/attack resilience in distributed SGD under a generalized Byzantine model.
• Develop aggregation rules that tolerate arbitrary per-dimension Byzantine values across workers.
• Prove convergence to critical points for the proposed rules.
• Evaluate computational efficiency and empirical robustness against realistic Byzantine attacks.

Proposed method

• Define generalized Byzantine model where per-dimension values can be corrupted across workers.
• Propose three aggregation rules: geometric median, marginal median, and mean around median.
• Prove dimensional Byzantine resilience for the proposed rules under q Byzantine per dimension.
• Analyze time complexity of each aggregation rule (nearly linear for the median-based methods).
• Show that these rules converge to critical points when resilience conditions hold.

Experimental results

Research questions

• RQ1 Can synchronous SGD converge to good solutions under a generalized Byzantine failure model where Byzantine values can occur at the per-dimension level across workers?
• RQ2 Do median-based aggregation rules provide dimensional Byzantine resilience, and under what conditions?
• RQ3 How do the proposed rules compare to classic Byzantine defenses in terms of convergence, robustness, and computational efficiency?
• RQ4 What is the impact of attacker models (Gaussian, omniscient, gambler) on convergence under the proposed aggregations?

Key findings

• Three median-based aggregations (geometric median, marginal median, mean around median) achieve convergence to critical points under dimensional Byzantine resilience.
• Marginal median and mean around median are proven to be dimensional Byzantine-resilient, enabling per-dimension defense against Byzantine values.
• Geometric median has classic Byzantine resilience with per-dimension considerations, and the methods have nearly linear time complexity in practice.
• Empirical results show the proposed methods outperform baseline aggregations under Gaussian, omniscient, bit-flip, and gambler attacks on MNIST and CIFAR-10 tasks.
• Mean-based methods are less robust than median-based methods under certain attacks, highlighting the advantage of marginal/mean-around-median under dimensional attacks.