[Paper Review] Byzantine-Robust Decentralized Learning via ClippedGossip

Lie He, Sai Praneeth Karimireddy+1 more|arXiv (Cornell University)|Feb 3, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes ClippedGossip, a Byzantine-robust decentralized learning algorithm that achieves provable convergence to an $\mathcal{O}(\delta_{\max}\zeta^{2}/\gamma^{2})$ neighborhood of a stationary point for non-convex objectives on arbitrary communication graphs. It combines local clipping with gossip averaging to defend against malicious workers while maintaining fast convergence rates even under adversarial conditions.

ABSTRACT

In this paper, we study the challenging task of Byzantine-robust decentralized training on arbitrary communication graphs. Unlike federated learning where workers communicate through a server, workers in the decentralized environment can only talk to their neighbors, making it harder to reach consensus and benefit from collaborative training. To address these issues, we propose a ClippedGossip algorithm for Byzantine-robust consensus and optimization, which is the first to provably converge to a $O(δ_{\max}ζ^2/γ^2)$ neighborhood of the stationary point for non-convex objectives under standard assumptions. Finally, we demonstrate the encouraging empirical performance of ClippedGossip under a large number of attacks.

Motivation & Objective

• To address the challenge of Byzantine-robust decentralized training in arbitrary communication graphs where workers communicate only with neighbors and no central server exists.
• To overcome limitations of prior methods that require dense connectivity, lack convergence guarantees, or assume honest majority in the network.
• To establish a new network robustness criterion based on spectral gap $\gamma$ and number of Byzantine workers $\delta_{\max}$, applicable to broader graph topologies.
• To provide the first provable convergence rate for robust decentralized stochastic non-convex optimization under standard assumptions.
• To improve convergence rates for non-robust decentralized SGD using local momentum.

Proposed method

• ClippedGossip operates in three stages per iteration: (1) workers collect neighbor models, (2) they clip neighbor models to lie within a radius $\tau_i^{t+1}$ around their own model to limit influence of outliers, and (3) they perform gossip averaging over the clipped models.
• The clipping step ensures that Byzantine updates, which may be arbitrarily large, are bounded and cannot dominate the consensus process.
• The method uses a mixing matrix to define the gossip averaging process, ensuring information spreads across the network while preserving robustness.
• A novel network robustness criterion is introduced, linking convergence to the spectral gap $\gamma$ of the communication graph and the maximum number of Byzantine workers $\delta_{\max}$.
• The algorithm is designed to work without requiring a global honest majority or secure broadcast, making it suitable for real-world decentralized systems.
• Local momentum is incorporated to achieve faster convergence rates in the non-robust setting, improving upon prior state-of-the-art.

Experimental results

Research questions

• RQ1 Can we design a decentralized learning algorithm that achieves provable convergence under Byzantine attacks on arbitrary communication graphs?
• RQ2 How does the spectral gap $\gamma$ of the communication graph and the number of Byzantine workers $\delta_{\max}$ jointly affect the robustness and convergence of decentralized learning?
• RQ3 Can we achieve faster convergence rates in non-robust decentralized SGD by incorporating local momentum?
• RQ4 How does ClippedGossip compare to prior methods in terms of convergence speed and robustness under realistic attack models?
• RQ5 Can the algorithm maintain consensus even when Byzantine workers are strategically placed to disrupt information flow?

Key findings

• ClippedGossip achieves provable convergence to an $\mathcal{O}(\delta_{\max}\zeta^{2}/\gamma^{2})$ neighborhood of a stationary point for non-convex objectives under standard assumptions.
• The method introduces a new robustness criterion based on the spectral gap $\gamma$ and $\delta_{\max}$, which applies to a broader class of graphs than prior works requiring dense connectivity or honest majority.
• ClippedGossip outperforms existing methods in both convergence speed and robustness, especially under dissensus attacks that amplify disagreement among honest workers.
• The algorithm maintains fast convergence even under strong Byzantine attacks, including those that exploit topology and gradient variance.
• The paper establishes the fastest known convergence rate for non-robust decentralized stochastic non-convex optimization: $\mathcal{O}(\frac{\sigma^{2}}{n\varepsilon^{2}} + \frac{\sigma^{2/3}}{\gamma^{2/3}\varepsilon^{4/3}})$ using local momentum.
• Empirical results show ClippedGossip converges reliably even when Byzantine workers are present, while prior methods like TM fail to reach consensus on common topologies like Dumbbell or under non-IID data.