[Paper Review] Optimal algorithms for smooth and strongly convex distributed optimization in networks

Kevin G. Seaman, Francis Bach+3 more|arXiv (Cornell University)|Feb 28, 2017

Distributed Control Multi-Agent Systems24 references189 citations

TL;DR

The paper derives optimal convergence rates for smooth and strongly convex distributed optimization in centralized and decentralized networks, and introduces SSDA and MSDA as optimal methods matching these bounds.

ABSTRACT

In this paper, we determine the optimal convergence rates for strongly convex and smooth distributed optimization in two settings: centralized and decentralized communications over a network. For centralized (i.e. master/slave) algorithms, we show that distributing Nesterov's accelerated gradient descent is optimal and achieves a precision $\varepsilon > 0$ in time $O(\sqrt{κ_g}(1+Δτ)\ln(1/\varepsilon))$, where $κ_g$ is the condition number of the (global) function to optimize, $Δ$ is the diameter of the network, and $τ$ (resp. $1$) is the time needed to communicate values between two neighbors (resp. perform local computations). For decentralized algorithms based on gossip, we provide the first optimal algorithm, called the multi-step dual accelerated (MSDA) method, that achieves a precision $\varepsilon > 0$ in time $O(\sqrt{κ_l}(1+\fracτ{\sqrtγ})\ln(1/\varepsilon))$, where $κ_l$ is the condition number of the local functions and $γ$ is the (normalized) eigengap of the gossip matrix used for communication between nodes. We then verify the efficiency of MSDA against state-of-the-art methods for two problems: least-squares regression and classification by logistic regression.

Motivation & Objective

• Determine the fundamental, optimal convergence rates for distributed optimization with smooth and strongly convex objectives in centralized and decentralized networks.
• Propose algorithms that achieve these optimal rates: distribute Nesterov’s acceleration for centralized settings and introduce dual-accelerated methods for decentralized settings.
• Analyze how network properties (diameter, eigengap) affect complexity bounds and compare to state-of-the-art methods.
• Validate the proposed methods on least-squares regression and logistic regression to illustrate practical efficiency.

Proposed method

• For centralized problems, distribute Nesterov’s accelerated gradient descent to achieve a rate O(sqrt(kappa_g) (1+Δτ) ln(1/ε)).
• For decentralized problems, formulate a dual problem and derive the single-step dual accelerated (SSDA) method achieving O(sqrt(kappa_l) (1+τ/√γ) ln(1/ε)) where γ is the normalized eigengap of the gossip matrix.
• Introduce a multi-step dual accelerated (MSDA) method by applying Chebyshev acceleration to the gossip step, yielding the rate O(sqrt(kappa_l) (1+τ/√γ) ln(1/ε)).
• Use a dual formulation where the problem is min over variables with equality constraints Theta√W=0 and solve via accelerated gradient on the dual.
• Apply Chebyshev polynomial acceleration P_K(W) to improve the effective gossip operator and achieve optimal dependence on γ with K ≈ ⌊1/√γ⌋.
• Discuss extensions like composite functions, warm starts, and asynchronous variations.

Experimental results

Research questions

• RQ1 What are the optimal oracle complexity lower bounds for centralized and decentralized distributed optimization with smooth and strongly convex objectives?
• RQ2 Can these lower bounds be achieved by practical algorithms in both centralized and decentralized settings?
• RQ3 How do network diameter Δ and gossip eigengap γ affect the convergence rate bounds?
• RQ4 How do SSDA and MSDA perform relative to existing methods (D-ADMM, EXTRA, DIGing) in practical problems like least-squares and logistic regression?

Key findings

• For centralized master/slave optimization, distributing Nesterov’s accelerated gradient descent achieves the optimal rate with time O(sqrt(kappa_g) (1+Δτ) ln(1/ε)).
• For decentralized (gossip-based) optimization, the MSDA method attains the optimal rate O(sqrt(kappa_l) (1+τ/√γ) ln(1/ε)).
• A lower bound shows any black-box procedure requires at least these orders of complexity in terms of kappa_g, Δ, τ (centralized) and kappa_l, γ, τ (decentralized).
• SSDA offers a simple dual-accelerated approach with a proved O((1+τ) sqrt(kappa_l/γ) ln(1/ε)) time to precision.
• MSDA improves the gossip step via Chebyshev acceleration to achieve the same order with better practical efficiency when communication is cheap, giving O(sqrt(kappa_l) (1+τ/√γ) ln(1/ε)).
• Experimental results on least-squares and logistic regression show DAGD is best among centralized methods, while MSDA is the strongest decentralized competitor, outperforming D-ADMM, EXTRA, and DIGing under various settings.