[论文解读] LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning

Tianyi Chen, Georgios B. Giannakis+2 more|arXiv (Cornell University)|May 24, 2018

Stochastic Gradient Optimization Techniques被引用 198

一句话总结

LAG 通过懒惰地重复使用梯度来减少分布式学习中的通信，在异质数据设置下实现与批量梯度下降相当的收敛速度，同时降低通信轮次。

ABSTRACT

This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation. Simple rules are designed to detect slowly-varying gradients and, therefore, trigger the reuse of outdated gradients. The resultant gradient-based algorithms are termed Lazily Aggregated Gradient --- justifying our acronym LAG used henceforth. Theoretically, the merits of this contribution are: i) the convergence rate is the same as batch gradient descent in strongly-convex, convex, and nonconvex smooth cases; and, ii) if the distributed datasets are heterogeneous (quantified by certain measurable constants), the communication rounds needed to achieve a targeted accuracy are reduced thanks to the adaptive reuse of lagged gradients. Numerical experiments on both synthetic and real data corroborate a significant communication reduction compared to alternatives.

研究动机与目标

• Motivate and develop communication-efficient gradient methods for distributed learning with multiple workers.
• Introduce lazy gradient aggregation to reduce per-iteration communication without harming convergence.
• Provide theoretical convergence guarantees under convex, strongly convex, and nonconvex smooth conditions.
• Quantify communication savings in heterogeneous data settings and identify when LAG outperforms standard GD.

提出的方法

• Formulate LAG as a lazy update of the GD step by reusing outdated gradients from workers unless a refinement is large.
• Define the LAG iteration where the gradient is updated as ∇^k = ∇^{k-1} + ∑_{m∈M^k} δ∇^k_m with δ∇^k_m = ∇L_m(θ^k) − ∇L_m(hatθ_m^{k-1}).
• Propose two implementation variants: LAG-WK (workers decide when to send updates) and LAG-PS (server decides which workers communicate).
• Derive descent lemmas for LAG (Lemmas 1 and 2) and establish a Lyapunov function V^k to analyze convergence.
• Provide iteration and communication complexity results and show conditions under which C_LAG(ε) < C_GD(ε) in heterogeneous settings.
• Discuss practical trigger rules based on gradients and recent iterates (LAG-WK condition and LAG-PS condition) to balance communication and convergence.

实验结果

研究问题

• RQ1 Can lazy gradient aggregation achieve similar convergence rates to batch GD under convex, strongly convex, and nonconvex smooth settings?
• RQ2 Under what heterogeneity conditions does LAG reduce communication rounds compared to traditional GD?
• RQ3 How do the proposed trigger rules (LAG-WK and LAG-PS) influence per-iteration descent and overall communication complexity?
• RQ4 What is the impact of data heterogeneity, via the heterogeneity score h(γ), on LAG's performance?

主要发现

• LAG achieves convergence rates with the same order as batch GD for strongly convex, convex, and nonconvex smooth cases.
• LAG can reduce communication rounds substantially in heterogeneous data settings by reusing lagged gradients.
• A quantifiable communication complexity bound shows potential C_LAG(ε) < C_GD(ε) when a sufficient fraction of workers have small local smoothness L_m.
• Two practical variants (LAG-WK and LAG-PS) provide comparable convergence guarantees with different communication strategies.
• Empirical results indicate significant communication reduction compared to alternatives, validating the theoretical benefits.