[Paper Review] GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning

Anis Elgabli, Jihong Park+3 more|arXiv (Cornell University)|Aug 30, 2019

Indoor and Outdoor Localization Technologies55 references49 citations

TL;DR

GADMM introduces a decentralized, two-neighbor communication scheme that splits workers into two groups to achieve fast convergence with reduced communication overhead, and extends to a Dynamic GADMM for time-varying networks.

ABSTRACT

When the data is distributed across multiple servers, lowering the communication cost between the servers (or workers) while solving the distributed learning problem is an important problem and is the focus of this paper. In particular, we propose a fast, and communication-efficient decentralized framework to solve the distributed machine learning (DML) problem. The proposed algorithm, Group Alternating Direction Method of Multipliers (GADMM) is based on the Alternating Direction Method of Multipliers (ADMM) framework. The key novelty in GADMM is that it solves the problem in a decentralized topology where at most half of the workers are competing for the limited communication resources at any given time. Moreover, each worker exchanges the locally trained model only with two neighboring workers, thereby training a global model with a lower amount of communication overhead in each exchange. We prove that GADMM converges to the optimal solution for convex loss functions, and numerically show that it converges faster and more communication-efficient than the state-of-the-art communication-efficient algorithms such as the Lazily Aggregated Gradient (LAG) and dual averaging, in linear and logistic regression tasks on synthetic and real datasets. Furthermore, we propose Dynamic GADMM (D-GADMM), a variant of GADMM, and prove its convergence under the time-varying network topology of the workers.

Motivation & Objective

• Motivate the need for communication-efficient distributed optimization in large-scale DML.
• Propose a decentralized ADMM-based framework (GADMM) that reduces per-iteration communication.
• Guarantee convergence to the optimal solution for convex losses under a two-group, two-neighbor communication topology.
• Introduce Dynamic GADMM (D-GADMM) to handle time-varying network topologies while preserving convergence.

Proposed method

• Formulate distributed learning as a consensus problem with local objectives f_n and a global parameter Theta.
• Develop GADMM by partitioning workers into head and tail groups and enforcing Theta_n = Theta_{n+1} with two-neighbor communication.
• Derive an augmented Lagrangian and update rules that allow head workers to update in parallel and tail workers to complete the iteration, ensuring convergence for convex problems.
• Prove convergence of GADMM to the optimal solution under convexity, via primal-dual residuals and a Lyapunov function.
• Extend the framework to time-varying networks with D-GADMM, which periodically refreshes neighbor relations and maintains convergence guarantees.

Experimental results

Research questions

• RQ1 Can a decentralized ADMM-based approach achieve the same optimality as centralized ADMM for convex loss functions?
• RQ2 How can communication be minimized in a distributed setting without central coordination while preserving convergence?
• RQ3 What is the impact of restricting each worker to two neighbors on convergence speed and communication overhead?
• RQ4 Can the framework adapt to time-varying network topologies and still converge?
• RQ5 Does randomizing neighbor connections (D-GADMM) improve convergence speed in static topologies?

Key findings

• GADMM converges to the optimal solution for convex loss functions with a two-neighbor, two-group communication pattern.
• The method achieves lower communication overhead than centralized ADMM and other baselines like LAG and dual averaging in linear and logistic regression tasks on synthetic and real data.
• D-GADMM extends GADMM to time-varying networks and inherits convergence guarantees, while potentially improving convergence speed by changing connectivity over time.
• Each iteration requires only a subset of workers to communicate, reducing uplink/downlink bottlenecks compared to fully centralized schemes.
• GADMM enables parallel updates within head and tail groups, accelerating practical convergence despite sparse communication.