[Paper Review] Quantized Decentralized Stochastic Learning over Directed Graphs

Hossein Taheri, Aryan Mokhtari+2 more|arXiv (Cornell University)|Feb 23, 2020

Cooperative Communication and Network Coding34 citations

TL;DR

The paper introduces quantized push-sum based decentralized learning over directed graphs, achieving the same convergence rates as exact-communication methods while significantly reducing communication. It provides gossip and stochastic optimization algorithms with provable convergence for convex and non-convex objectives.

ABSTRACT

We consider a decentralized stochastic learning problem where data points are distributed among computing nodes communicating over a directed graph. As the model size gets large, decentralized learning faces a major bottleneck that is the heavy communication load due to each node transmitting large messages (model updates) to its neighbors. To tackle this bottleneck, we propose the quantized decentralized stochastic learning algorithm over directed graphs that is based on the push-sum algorithm in decentralized consensus optimization. More importantly, we prove that our algorithm achieves the same convergence rates of the decentralized stochastic learning algorithm with exact-communication for both convex and non-convex losses. Numerical evaluations corroborate our main theoretical results and illustrate significant speed-up compared to the exact-communication methods.

Motivation & Objective

• Motivate decentralized learning when data is distributed across nodes connected by a directed graph.
• Propose quantized, communication-efficient algorithms based on push-sum for gossip and decentralized SGD.
• Establish convergence guarantees that match exact-communication methods for both convex and non-convex losses.
• Demonstrate significant communication speed-ups through numerical experiments.

Proposed method

• Extend the push-sum consensus framework to directed graphs with column-stochastic weights.
• Introduce quantization of transmitted updates to reduce communication load while maintaining convergence.
• Develop Algorithm 1 for Quantized Gossip over Directed Graphs and Algorithm 2 for Quantized Decentralized SGD over Directed Graphs.
• Prove convergence results: Gossip with quantization converges at a rate governed by lambda and delta, matching exact push-sum up to quantization error.
• Show that for convex objectives, the time-averaged objective converges at rate O(1/sqrt(nT)); for non-convex objectives, the squared gradient norm converges at rate O(1/sqrt(nT)).
• Provide numerical experiments comparing quantized versus exact-communication push-sum, highlighting bit-wise communication speed-ups and performance.

Experimental results

Research questions

• RQ1 Can quantized communication in push-sum-based decentralized optimization over directed graphs achieve the same convergence rates as exact-communication methods?
• RQ2 What are the convergence guarantees for gossip and decentralized SGD when using quantized messages on strongly connected directed graphs?
• RQ3 How does quantization affect the rate constants and practical communication efficiency in both convex and non-convex settings?
• RQ4 Do directed-graph topologies and quantization noise impact convergence rates, and by how much?
• RQ5 What is the trade-off between number of bits transmitted and convergence speed in realistic experiments?

Key findings

• Quantized gossip achieves convergence to the initial mean with rate depending on graph parameters and quantization level.
• For convex objectives, the proposed quantized decentralized SGD attains a convergence rate of O(1/√(nT)) for the averaged objective.
• For non-convex objectives, the gradient norm converges to zero at rate O(1/√(nT)) under the proposed scheme.
• Quantization noise vanishes asymptotically and yields substantial communication savings compared to exact-communication push-sum (up to 5–10x in experiments).
• Experiments show comparable performance to exact methods per iteration on simpler graphs, with significant reductions in total bits transmitted.
• Theoretical results demonstrate that the rate bounds match those of exact communication up to terms influenced by quantization and graph structure.