[Paper Review] signSGD with Majority Vote is Communication Efficient And Fault Tolerant

Jeremy Bernstein, Jiawei Zhao+2 more|arXiv (Cornell University)|Oct 11, 2018

Retinal Imaging and Analysis125 citations

TL;DR

The paper introduces signSGD with majority vote, a 1-bit communication, robust distributed optimization method that converges under noise and tolerates up to 50% Byzantine workers, with empirical speedups on large-scale tasks.

ABSTRACT

Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines. As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable. The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults. We explore a particularly simple algorithm for robust, communication-efficient learning---signSGD. Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote. This algorithm uses $32 imes$ less communication per iteration than full-precision, distributed SGD. Under natural conditions verified by experiment, we prove that signSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of Adam as a byproduct. Aggregating sign gradients by majority vote means that no individual worker has too much power. We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially. The class of adversaries we consider includes as special cases those that invert or randomise their gradient estimate. On the practical side, we built our distributed training system in Pytorch. Benchmarking against the state of the art collective communications library (NCCL), our framework---with the parameter server housed entirely on one machine---led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines.

Motivation & Objective

• Motivate the need for fast, robust, and communication-efficient distributed training for large neural networks.
• Propose a simple gradient-sign based update with majority vote to compress communication and increase fault tolerance.
• Provide convergence theory under realistic assumptions for mini-batch training and a Byzantine fault model.
• Demonstrate practical benefits and trade-offs through large-scale experiments on ImageNet and language models.

Proposed method

• Workers compute mini-batch gradients and send only the sign of their momentum to a parameter server.
• The server aggregates signs via majority vote and broadcasts the majority sign back to workers.
• Update rule uses x <- x - eta * (sign(V) + lambda x) with momentum on each worker.
• Provide a theoretical convergence analysis under non-convex objectives with unimodal, symmetric gradient noise (Assumptions 1-4).
• Show robustness to blind multiplicative adversaries with fraction alpha < 1/2 of workers being adversarial (Theorem 2).
• Implement Signum in PyTorch with 1-bit tensor compression and compare to NCCL in large-scale experiments.

Experimental results

Research questions

• RQ1 Does signSGD with majority vote converge in non-convex settings under realistic gradient noise assumptions?
• RQ2 How does majority voting affect robustness to Byzantine/adversarial worker behavior?
• RQ3 What communication and wall-clock performance advantages does the method offer on large-scale datasets compared to full-precision SGD and other compression schemes?

Key findings

• Theoretical convergence rate for mini-batch signSGD matches SGD in O(1/sqrt(N)) under the stated assumptions.
• Majority vote provides Byzantine fault tolerance up to 50% adversarial workers with degraded but guaranteed convergence.
• Empirical results show up to 25% wall-clock time speedup on Imagenet training with 7–15 AWS p3.2xlarge machines, with slight generalisation degradation.
• Signum with majority vote can outperform some QSGD variants in communication cost while maintaining competitive convergence.
• Gradient noise in practice is often unimodal and symmetric, supporting the assumptions and convergence of the method.