[Paper Review] Byzantine Fault Tolerance in Distributed Machine Learning : a Survey

Djamila Bouhata, Hamouma Moumen+2 more|arXiv (Cornell University)|May 5, 2022

Blockchain Technology Applications and Security4 citations

TL;DR

This survey provides a comprehensive analysis of Byzantine Fault Tolerance (BFT) in distributed machine learning (DML), focusing on first-order optimization methods like Stochastic Gradient Descent (SGD). It classifies BFT approaches based on communication patterns, optimization methods, and topology, highlighting filtering schemes as most prevalent and blockchain-based solutions as promising despite limited adoption.

ABSTRACT

Byzantine Fault Tolerance (BFT) is one of the most challenging problems in Distributed Machine Learning (DML), defined as the resilience of a fault-tolerant system in the presence of malicious components. Byzantine failures are still difficult to deal with due to their unrestricted nature, which results in the possibility of generating arbitrary data. Significant research efforts are constantly being made to implement BFT in DML. Some recent studies have considered various BFT approaches in DML. However, some aspects are limited, such as the few approaches analyzed, and there is no classification of the techniques used in the studied approaches. In this paper, we present a survey of recent work surrounding BFT in DML, mainly in first-order optimization methods, especially Stochastic Gradient Descent(SGD). We highlight key techniques as well as fundamental approaches. We provide an illustrative description of the techniques used in BFT in DML, with a proposed classification of BFT approaches in the context of their fundamental techniques. This classification is established on specific criteria such as communication process, optimization method, and topology setting, which characterize future work methods addressing open challenge

Motivation & Objective

• To analyze recent approaches to Byzantine Fault Tolerance (BFT) in distributed machine learning (DML).
• To classify BFT techniques based on communication process, optimization method, and topology settings.
• To identify dominant techniques such as filtering schemes and emerging blockchain-based solutions.
• To highlight open challenges in BFT for DML, including node failures, adversarial behavior, and communication inefficiencies.
• To guide future research by outlining key directions, especially in federated learning and partially asynchronous settings.

Proposed method

• The survey conducts a systematic review of recent works on BFT in DML, focusing on first-order optimization methods, particularly SGD.
• It proposes a classification framework for BFT approaches based on three criteria: communication process (synchronous, asynchronous, partially asynchronous), optimization method (first-order, second-order), and topology (centralized, decentralized, fully distributed).
• It evaluates techniques such as filtering schemes (e.g., norm filtering, norm-cap filtering), coding schemes (e.g., error-correcting codes), and blockchain-based systems (e.g., LearningChain, BlockDeepNet).
• The analysis compares the computational complexity, convergence behavior, and data privacy properties of each approach.
• It evaluates the performance of BFT mechanisms under various failure models, including Byzantine workers generating arbitrary gradients.
• It discusses the integration of BFT with modern DML paradigms like federated learning, emphasizing communication efficiency and resilience to node dropout.

Experimental results

Research questions

• RQ1 What are the dominant techniques used in BFT for distributed machine learning, and how do they compare in terms of resilience and efficiency?
• RQ2 How do different communication models—synchronous, asynchronous, and partially asynchronous—affect the design and performance of BFT mechanisms in DML?
• RQ3 What is the role of filtering schemes, coding schemes, and blockchain in mitigating Byzantine attacks in DML, and how do they differ in implementation and scalability?
• RQ4 How do topology settings (centralized, decentralized, peer-to-peer) influence the effectiveness and deployment of BFT solutions in DML?
• RQ5 What are the key open challenges in achieving robust BFT in real-world DML systems, especially in federated learning with heterogeneous and unreliable nodes?

Key findings

• Filtering schemes are the most widely used BFT technique in current DML research, outperforming coding schemes and blockchain in terms of adoption and implementation ease.
• Synchronous training settings are preferred in most studied works, despite their limitations in real-world deployment due to strict coordination requirements.
• Blockchain-based BFT solutions, though limited in number, show strong potential in enabling trustless, collaborative model training with data privacy and resistance to poisoning attacks.
• The integration of blockchain with other BFT techniques can enhance security and scalability, offering a path toward resilient, decentralized DML systems.
• Partially asynchronous and asynchronous training models remain underexplored, representing a key gap in current BFT research for DML.
• Federated learning is identified as a critical future research area for BFT, due to its vulnerability to Byzantine nodes and unreliable communication in edge environments.