[Paper Review] Accelerated Decentralized Optimization with Local Updates for Smooth and Strongly Convex Objectives

Hadrien Hendrikx, Francis Bach+1 more|arXiv (Cornell University)|Oct 5, 2018

Distributed Control Multi-Agent Systems7 references17 citations

TL;DR

This paper proposes ESDACD, a decentralized accelerated optimization algorithm for smooth and strongly convex functions that uses local synchrony and edge-based updates. It achieves convergence rates matching the optimal synchronous SSDA algorithm while enabling asynchronous execution, with provably improved convergence for the second moment of error in gossip settings, especially in heterogeneous networks.

ABSTRACT

In this paper, we study the problem of minimizing a sum of smooth and strongly convex functions split over the nodes of a network in a decentralized fashion. We propose the algorithm $ESDACD$, a decentralized accelerated algorithm that only requires local synchrony. Its rate depends on the condition number $κ$ of the local functions as well as the network topology and delays. Under mild assumptions on the topology of the graph, $ESDACD$ takes a time $O((τ_{\max} + Δ_{\max})\sqrt{κ/γ}\ln(ε^{-1}))$ to reach a precision $ε$ where $γ$ is the spectral gap of the graph, $τ_{\max}$ the maximum communication delay and $Δ_{\max}$ the maximum computation time. Therefore, it matches the rate of $SSDA$, which is optimal when $τ_{\max} = Ω\left(Δ_{\max} ight)$. Applying $ESDACD$ to quadratic local functions leads to an accelerated randomized gossip algorithm of rate $O( \sqrt{θ_{ m gossip}/n})$ where $θ_{ m gossip}$ is the rate of the standard randomized gossip. To the best of our knowledge, it is the first asynchronous gossip algorithm with a provably improved rate of convergence of the second moment of the error. We illustrate these results with experiments in idealized settings.

Motivation & Objective

• To design a decentralized optimization algorithm that achieves accelerated convergence rates comparable to synchronous methods while requiring only local synchrony.
• To address the limitations of centralized architectures, such as communication bottlenecks and single-point failures, in large-scale distributed learning.
• To improve convergence speed in decentralized settings with heterogeneous node capabilities and varying local condition numbers.
• To develop an asynchronous gossip algorithm with provably better convergence rates than standard randomized gossip, particularly in terms of second moment of error.
• To demonstrate that local parameter tuning and edge-specific updates can enhance performance in heterogeneous networks without sacrificing convergence guarantees.

Proposed method

• ESDACD is based on accelerated dual coordinate descent, using edge-sampling to update neighboring nodes asynchronously.
• The algorithm performs local gradient updates and global contraction steps via a randomized gossip mechanism on edges.
• It introduces edge-specific step sizes and weights that adapt to local smoothness and communication delays.
• The method leverages Nesterov-style acceleration in the dual formulation to achieve faster convergence.
• Updates are performed in the order they are sampled per node, ensuring local synchrony without requiring global coordination.
• The algorithm is applied to both general smooth and strongly convex optimization and the distributed average consensus problem.

Experimental results

Research questions

• RQ1 Can a decentralized optimization algorithm achieve convergence rates matching optimal synchronous methods like SSDA while requiring only local synchrony?
• RQ2 Does an asynchronous gossip algorithm with local updates and edge-specific parameters outperform standard randomized gossip in terms of second moment of error?
• RQ3 How does ESDACD perform in heterogeneous networks with varying local condition numbers and computation delays?
• RQ4 Can local parameter tuning in ESDACD adaptively improve convergence speed in non-uniform settings?
• RQ5 What is the impact of communication delays and computation time on the convergence rate of decentralized accelerated algorithms?

Key findings

• ESDACD achieves a convergence rate of $ O((\tau_{\max}+\Delta_{\max})\sqrt{\kappa/\gamma}\ln(\epsilon^{-1})) $, matching the optimal rate of SSDA under mild graph assumptions.
• In homogeneous settings, ESDACD is roughly two times slower than SSDA per iteration, but uses 2× fewer gradients and 8× fewer messages on a grid graph.
• In heterogeneous settings with variable local condition numbers, ESDACD achieves significantly lower final error than SSDA despite using half the number of gradients.
• For the distributed average consensus problem, ESDACD yields the first asynchronous gossip algorithm with a provably improved rate of convergence for the second moment of error.
• The algorithm adapts well to local variations in smoothness and computation speed, outperforming SSDA in scenarios with high variance in node capabilities.
• Empirical results show that ESDACD completes two iterations in the time SSDA completes one, indicating superior computational efficiency in heterogeneous environments.