[Paper Review] Secure Byzantine-Robust Machine Learning

Lie He, Sai Praneeth Karimireddy+1 more|arXiv (Cornell University)|Jun 8, 2020

Privacy-Preserving Technologies in DataComputer Science45 references39 citations

TL;DR

Proposes a secure two-server aggregation framework that achieves input privacy and Byzantine-robustness for distributed learning, with fault tolerance and compatibility with local differential privacy and existing robust rules.

ABSTRACT

Increasingly machine learning systems are being deployed to edge servers and devices (e.g. mobile phones) and trained in a collaborative manner. Such distributed/federated/decentralized training raises a number of concerns about the robustness, privacy, and security of the procedure. While extensive work has been done in tackling with robustness, privacy, or security individually, their combination has rarely been studied. In this paper, we propose a secure two-server protocol that offers both input privacy and Byzantine-robustness. In addition, this protocol is communication-efficient, fault-tolerant and enjoys local differential privacy.

Motivation & Objective

• Tackle privacy and robustness simultaneously in distributed/federated learning.
• Develop a secure aggregation protocol that preserves input privacy against honest-but-curious servers.
• Enable compatibility with existing Byzantine-robust aggregation rules while maintaining exactness.
• Ensure fault tolerance, scalability, and low communication overhead in practical deployments.

Proposed method

• Workers secret-share their updates with two non-colluding servers.
• Two-server 2PC-based secure aggregation computes either standard sum or a robust, distance-based aggregation using Beaver’s triples.
• For robustness, pairwise distances between updates are computed securely on the servers and fed to a robust aggregation oracle to select weights.
• The final aggregated update is reconstructed and applied to the public model, with exactness preserved relative to non-private robust methods.
• The protocol supports dropout/new joiners of workers and is designed to be communication-efficient (uplink within a factor of 2 of non-private).
• The framework can be combined with differential privacy, enabling locally differentially private mechanisms when integrated with DP-based training.

Experimental results

Research questions

• RQ1 Can input privacy be achieved in a Byzantine-robust distributed learning setting without leaking individual updates?
• RQ2 How can distance-based robust aggregation rules be securely integrated into a two-server MPC framework?
• RQ3 Does the secure protocol preserve the exact results of non-private robust aggregation?
• RQ4 What are the communication and fault-tolerance properties of the proposed scheme in practice?
• RQ5 How does the approach interface with differential privacy in practice?

Key findings

• The proposed two-server secure aggregation yields the exact same result as the non-privacy-preserving robust aggregation (exactness).
• The protocol preserves strong input privacy, such that servers learn only the final aggregated update and, in robustness mode, pairwise distances between updates, not the updates themselves.
• The approach is fault-tolerant to worker dropouts and new joiners, with a one-round communication scheme and modest overhead.
• The communication overhead is within a factor of 2 of non-private methods, and server-to-server communication can be accelerated; robustness can be achieved with existing rules like Krum.
• The method is compatible with local differential privacy and can be integrated with DP-based training techniques.