[论文解读] A Unified Theory of Decentralized SGD with Changing Topology and Local Updates

Anastasia Koloskova, Nicolas Loizou+3 more|arXiv (Cornell University)|Mar 23, 2020

Stochastic Gradient Optimization Techniques参考文献 92被引用 49

一句话总结

该论文提出了一个统一的分布式 SGD 收敛性分析，针对带本地更新和时变、随机 Gossip 拓扑的情形，给出可以插值 iid 与异质数据的普适速率，并在过参数化情况下实现线性收敛。

ABSTRACT

Decentralized stochastic optimization methods have gained a lot of attention recently, mainly because of their cheap per iteration cost, data locality, and their communication-efficiency. In this paper we introduce a unified convergence analysis that covers a large variety of decentralized SGD methods which so far have required different intuitions, have different applications, and which have been developed separately in various communities. Our algorithmic framework covers local SGD updates and synchronous and pairwise gossip updates on adaptive network topology. We derive universal convergence rates for smooth (convex and non-convex) problems and the rates interpolate between the heterogeneous (non-identically distributed data) and iid-data settings, recovering linear convergence rates in many special cases, for instance for over-parametrized models. Our proofs rely on weak assumptions (typically improving over prior work in several aspects) and recover (and improve) the best known complexity results for a host of important scenarios, such as for instance coorperative SGD and federated averaging (local SGD).

研究动机与目标

• Develop a unified framework for gossip-based decentralized SGD that encompasses local updates and adaptive network topologies.
• Derive universal convergence rates for smooth convex and non-convex objectives under weak noise and heterogeneity assumptions.
• Show rate interpolation between iid and non-iid data settings and identify conditions yielding linear convergence in overparametrized regimes.
• Provide lower bounds demonstrating tightness of rates in strongly convex settings.
• Empirically verify the theoretical results and illustrate the impact of noise and data diversity on convergence.

提出的方法

• Model decentralized SGD with local gradient updates followed by a consensus (gossip) averaging step.
• Allow mixing matrices W(t) drawn from time-varying distributions and define a new expected consensus rate Assumption 4.
• Introduce a novel assumption on the expected mixing rate over τ consecutive steps (Assumption 4) to bound ||XWℓ,τ − X̄||F in expectation.
• Provide a unified convergence analysis that yields rates for non-convex, convex, and strongly convex settings (Theorem 2).
• Establish a lower bound (Theorem 3) showing necessity of heterogeneity terms for strongly convex cases.
• Relate the framework to special cases like Local SGD, Cooperative SGD, and periodic decentralized SGD (Section 5).

实验结果

研究问题

• RQ1 How can decentralized SGD with local updates and changing topology be analyzed under a unified framework?
• RQ2 What are the universal convergence rates for smooth convex and non-convex problems under heterogeneous data and time-varying gossip topologies?
• RQ3 Under what conditions do decentralized, non-centralized SGD methods achieve linear convergence in overparameterized settings?
• RQ4 How do noise and data diversity affect convergence, and are the rates tight?
• RQ5 Can existing decentralized SGD variants (e.g., Local SGD, periodic decentralized SGD) be recovered as special cases within the proposed framework?

主要发现

• The framework yields universal convergence rates for non-convex, convex, and strongly convex objectives under weak assumptions on noise and data diversity.
• Rates interpolate between heterogeneous (non-identically distributed) and iid data settings, and linear convergence is recovered in overparametrized scenarios.
• A lower bound shows the dependence on data heterogeneity is necessary in strong convexity, confirming tightness of the results.
• The analysis handles time-varying, randomly sampled mixing matrices and does not require per-step connectivity, only a cumulative mixing property (Assumption 4).
• The results specialize to and improve upon existing analyses for Local SGD and other decentralized schemes, under weaker or more general assumptions.
• Empirical results validate the tightness of the theoretical bounds and illustrate how noise and diversity influence convergence.