[論文レビュー] Optimal Algorithms for Non-Smooth Distributed Optimization in Networks

Kevin Scaman, Francis Bach+3 more|arXiv (Cornell University)|Jun 1, 2018

Distributed Control Multi-Agent Systems参考文献 11被引用数 79

ひとこと要約

この論文は、2つの規則性仮定（グローバルリップシッツとローカルリップシッツ）の下で、非スムーズな分散凸最適化の最適収束速度を導出します。ローカル規則性の下で MSPD、グローバル規則性の下で DRS を導入し、下界と次元依存の平滑化アプローチを提供します。

ABSTRACT

In this work, we consider the distributed optimization of non-smooth convex functions using a network of computing units. We investigate this problem under two regularity assumptions: (1) the Lipschitz continuity of the global objective function, and (2) the Lipschitz continuity of local individual functions. Under the local regularity assumption, we provide the first optimal first-order decentralized algorithm called multi-step primal-dual (MSPD) and its corresponding optimal convergence rate. A notable aspect of this result is that, for non-smooth functions, while the dominant term of the error is in $O(1/\sqrt{t})$, the structure of the communication network only impacts a second-order term in $O(1/t)$, where $t$ is time. In other words, the error due to limits in communication resources decreases at a fast rate even in the case of non-strongly-convex objective functions. Under the global regularity assumption, we provide a simple yet efficient algorithm called distributed randomized smoothing (DRS) based on a local smoothing of the objective function, and show that DRS is within a $d^{1/4}$ multiplicative factor of the optimal convergence rate, where $d$ is the underlying dimension.

研究の動機と目的

• Motivate distributed optimization of a non-smooth convex objective over a network of computing units.
• Derive optimal convergence rates under two regularity assumptions: global Lipschitz and local Lipschitz.
• Provide algorithms that achieve these optimal rates: MSPD for local regularity and DRS for global regularity.
• Establish lower bounds showing optimality of the proposed methods and discuss communication versus computation trade-offs.

提案手法

• Model the problem as minimizing the average of local convex functions over a strongly connected graph.
• Under local regularity, formulate the problem as a saddle-point and design the multi-step primal-dual (MSPD) algorithm with accelerated gossip to achieve optimal rates.
• Under global regularity, apply a distributed smoothing approach (DRS) based on Gaussian smoothing to obtain fast communication rates and analyze its convergence.
• Prove lower bounds that match the MSPD rates under local regularity and show DRS is within a d^{1/4} factor of optimal under global regularity.
• Extend the decentralized method with Chebyshev acceleration to reach optimal communication rates in MSPD.

実験結果

リサーチクエスチョン

• RQ1 What are the optimal convergence rates for non-smooth distributed optimization under global Lipschitz regularity?
• RQ2 What are the optimal convergence rates under local Lipschitz regularity, and can we design algorithms that achieve them?
• RQ3 How does network topology and communication affect the rates in non-smooth distributed optimization?
• RQ4 Can smoothing techniques yield dimension-dependent but near-optimal rates in the distributed setting?
• RQ5 What are the fundamental lower bounds for computation and communication in decentralized non-smooth optimization?

主な発見

• DRS achieves an approximation error of at most ε in time bounded by O(RL_g/ε · (Δτ+1) d^{1/4} + (RL_g/ε)^2) under global regularity.
• MSPD is optimal under local regularity, with time to ε-approximation bounded by O(RL_ℓ/ε · τ/√γ(W) + (RL_ℓ/ε)^2).
• Under local regularity, the dominant error term is O(1/√t) from local computations while the communication error decays as O(1/t).
• The lower bound shows that the DRS rate is optimal with respect to computation time and within a d^{1/4} factor of the optimal communication rate.
• MSPD achieves the optimal convergence rate by incorporating accelerated gossip and a primal-dual update scheme.