[Paper Review] Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance

Cong Xie, Oluwasanmi Koyejo+1 more|arXiv (Cornell University)|May 25, 2018

Stochastic Gradient Optimization TechniquesComputer Science83 citations

TL;DR

Zeno introduces a suspicion-based aggregation rule for distributed SGD that tolerates arbitrarily many faulty workers, requiring only at least one non-faulty worker, and proves convergence for non-convex objectives.

ABSTRACT

We present Zeno, a technique to make distributed machine learning, particularly Stochastic Gradient Descent (SGD), tolerant to an arbitrary number of faulty workers. Zeno generalizes previous results that assumed a majority of non-faulty nodes; we need assume only one non-faulty worker. Our key idea is to suspect workers that are potentially defective. Since this is likely to lead to false positives, we use a ranking-based preference mechanism. We prove the convergence of SGD for non-convex problems under these scenarios. Experimental results show that Zeno outperforms existing approaches.

Motivation & Objective

• Motivate fault-tolerant distributed SGD beyond majority-fault assumptions.
• Develop a suspicion-based aggregation mechanism to handle Byzantine-like faults.
• Prove convergence of SGD on non-convex objectives under arbitrary fault patterns.
• Demonstrate empirical robustness and practicality on standard benchmarks and disjoint data settings.

Proposed method

• Represent each candidate gradient as a suspect and score it using a stochastic zero-order oracle.
• Define a stochastic descendant score combining estimated loss descent and update magnitude.
• Rank gradient estimators by score and aggregate the top m-b estimates with Zeno_b.
• Prove convergence guarantees showing identical asymptotic rate to fault-free distributed SGD under certain conditions.
• Analyze time complexity and compare with baselines like Mean, Median, and Krum.
• Extend analysis to disjoint non-identically distributed data across workers.

Experimental results

Research questions

• RQ1 Can SGD converge for non-convex objectives when an arbitrary number of workers are faulty, as long as at least one is non-faulty?
• RQ2 Does a suspicion-based aggregation (Zeno) achieve convergence rates comparable to fault-free SGD under Byzantine-like faults?
• RQ3 How does Zeno perform empirically compared to existing robust aggregations under various fault models and data distributions?

Key findings

• Zeno converges with the same asymptotic time complexity as distributed synchronous SGD without faults.
• Empirical results show Zeno outperforms baselines when faults are present, including label-flipping and bit-flipping scenarios.
• Zeno remains effective with disjoint local data across workers and is robust to hyperparameter choices such as n_r, ρ, and b.
• The approach tolerates more than half faulty gradients and remains stable where some baselines fail.
• Zeno’s performance approaches that of fault-free SGD when faults are not dominant, and can outperform Krum in several fault scenarios.