[Paper Review] SLSGD: Secure and Efficient Distributed On-device Machine Learning

Cong Xie, Sanmi Koyejo+1 more|arXiv (Cornell University)|Mar 16, 2019

Privacy-Preserving Technologies in Data30 references4 citations

TL;DR

SLSGD is a secure, communication-efficient distributed on-device learning algorithm that uses local stochastic gradient descent with robust aggregation and moving averages to tolerate data poisoning and converge under non-IID and heterogeneous settings. It achieves provable convergence and stabilizes training even when a small number of workers are compromised.

ABSTRACT

We consider distributed on-device learning with limited communication and security requirements. We propose a new robust distributed optimization algorithm with efficient communication and attack tolerance. The proposed algorithm has provable convergence and robustness under non-IID settings. Empirical results show that the proposed algorithm stabilizes the convergence and tolerates data poisoning on a small number of workers.

Motivation & Objective

• Address the challenges of limited communication, heterogeneous computation, and non-IID data in on-device machine learning.
• Develop a distributed optimization algorithm that maintains convergence and robustness under data poisoning attacks.
• Provide theoretical convergence guarantees for non-IID and heterogeneous settings, where prior work relies on IID or homogeneous assumptions.
• Stabilize training through robust aggregation and moving average techniques to mitigate error accumulation and variance from infrequent synchronization.

Proposed method

• Uses local SGD with multiple local iterations per global update to reduce communication overhead.
• Employs robust aggregation via trimmed mean to filter out abnormal models from potentially poisoned workers.
• Applies a moving average to the global model update to reduce variance and error from infrequent synchronization.
• Activates a random subset of workers per global epoch to balance load and reduce communication frequency.
• Introduces adaptive learning rate decay (α) and trimming threshold (b) to control convergence stability and robustness.
• Theoretical analysis proves convergence for strongly convex, non-strongly convex, and restricted non-convex functions under non-IID and heterogeneous settings.

Experimental results

Research questions

• RQ1 Can local SGD with limited communication and robust aggregation achieve provable convergence under non-IID data distributions?
• RQ2 How does SLSGD tolerate data poisoning when a small number of workers are compromised?
• RQ3 What is the impact of local update count and hyperparameters (α, b) on convergence speed and stability?
• RQ4 How does unbalanced data partitioning affect convergence, and can SLSGD mitigate this variance?
• RQ5 Can robust aggregation and moving average effectively prevent model divergence under poisoning attacks?

Key findings

• SLSGD achieves provable convergence for strongly convex, non-strongly convex, and a restricted class of non-convex functions under non-IID and heterogeneous settings.
• SLSGD with trimmed mean (Option II) successfully prevents catastrophic divergence when up to 40% of data in each epoch is poisoned.
• Larger values of b (trimming threshold) and smaller values of α (learning rate) improve robustness and convergence speed under poisoning and unbalanced data.
• More local iterations per global epoch accelerate convergence, as shown in experiments with 3 full passes of local data (pass=3) leading to faster training loss reduction.
• In the absence of attacks, SLSGD with α=1 and b=0 (equivalent to FedAvg) converges fastest, but the performance gap with other configurations is minimal.
• Under unbalanced partitions and attacks, smaller α provides greater stabilization, and trimmed mean is essential to prevent divergence.