[Paper Review] Byzantine Stochastic Gradient Descent

Dan Alistarh, Zeyuan Allen-Zhu+1 more|arXiv (Cornell University)|Mar 23, 2018

Stochastic Gradient Optimization Techniques27 references104 citations

TL;DR

This paper develops Byzantine-robust SGD methods for distributed stochastic optimization, achieving near-optimal sample and time complexity under a fraction of Byzantine workers, with formal lower bounds. It provides algorithms that tolerate α < 1/2 Byzantine machines and match non-Byzantine performance up to additive α-dependent terms.

ABSTRACT

This paper studies the problem of distributed stochastic optimization in an adversarial setting where, out of the $m$ machines which allegedly compute stochastic gradients every iteration, an $α$-fraction are Byzantine, and can behave arbitrarily and adversarially. Our main result is a variant of stochastic gradient descent (SGD) which finds $\varepsilon$-approximate minimizers of convex functions in $T = ilde{O}\big( \frac{1}{\varepsilon^2 m} + \frac{α^2}{\varepsilon^2} \big)$ iterations. In contrast, traditional mini-batch SGD needs $T = O\big( \frac{1}{\varepsilon^2 m} \big)$ iterations, but cannot tolerate Byzantine failures. Further, we provide a lower bound showing that, up to logarithmic factors, our algorithm is information-theoretically optimal both in terms of sampling complexity and time complexity.

Motivation & Objective

• Motivate robust distributed optimization when some workers may act adversarially (Byzantine).
• Minimize a convex (or strongly convex) objective using stochastic gradients from multiple workers while tolerating Byzantine faults.
• Achieve information-theoretically optimal sample and time complexity up to logarithmic factors in the presence of Byzantine workers.
• Ensure scalability and low per-iteration communication in a Byzantine setting.

Proposed method

• Propose ByzantineSGD, a robust aggregation strategy that maintains two estimation sequences (A_i and B_i) per worker to detect and exclude Byzantine machines on the fly.
• Use medians (Amed, Bmed, ∇med) to identify a subset of good machines at each iteration without restarting the run.
• Update the iterate using the average gradient from the identified good set, mitigating Byzantine influence.
• Provide formal concentration lemmas (Event A, Event B, Event C) to bound the bias and variance introduced by Byzantine workers.
• Derive convergence guarantees for both non-strongly convex and strongly convex objectives, including smooth and non-smooth cases.
• Offer an epoch-based extension (ByzantineSGD in strong convexity) that repeats ByzantineSGD to achieve ε-optimality with a logarithmic number of epochs.

Experimental results

Research questions

• RQ1 How many iterations and samples are needed to reach an ε-approximate minimizer when a fraction α of workers are Byzantine?
• RQ2 Can we design SGD-like algorithms that tolerate Byzantine workers while maintaining near-optimal sample and time complexity?
• RQ3 What concentration tools and robust aggregation schemes enable reliable identification of good workers in the presence of adversarial behavior?
• RQ4 How do robustness guarantees differ between smooth vs non-smooth and convex vs strongly convex objectives?

Key findings

• ByzantineSGD achieves ε-approximate minimizers in T = e^{O(1/ε^2m) + α^2/ε^2} iterations for smooth convex objectives, and T = e^{O(1/σ + 1/(σεm) + α^2/σε)} for σ-strongly convex objectives.
• There is a matching information-theoretic lower bound showing the α-dependent term α^2/ε^2 (or α^2/(σε)) is necessary.
• The additive α-dependent term is unavoidable, while the remaining terms match non-Byzantine SGD up to constants, preserving parallel speedups up to α-related limits.
• The Byzantine fraction α < 1/2 is supported, and the algorithms maintain per-iteration communication efficiency by requiring only a single gradient from each worker.
• Compared to prior work (e.g., coordinate-wise median methods), ByzantineSGD achieves better sample and time scaling, especially in high dimensions, with fewer stochastic gradient computations.
• For strongly convex objectives, an epoch-based variant yields convergence in T = e^{O(L/σ) + V^2/(mσε) + α^2V^2/σε} iterations, aligning with standard GD rates plus Byzantine robustness.