[论文解读] CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression

Zhize Li, Peter Richtárik|arXiv (Cornell University)|Jul 20, 2021

Stochastic Gradient Optimization Techniques参考文献 41被引用 6

一句话总结

本文提出CANITA，一种新颖的压缩与加速梯度方法，用于分布式凸优化，结合了通信压缩与Nesterov风格的加速。通过利用ANITA框架并引入一种新分析方法，CANITA在有利条件下实现了更快的收敛速率$O\left(\sqrt{\frac{L}{\epsilon}}\right)$，显著优于现有非加速方法在压缩分布式问题中的最先进速率$O\left(\frac{L}{\epsilon}\right)$。

ABSTRACT

Due to the high communication cost in distributed and federated learning, methods relying on compressed communication are becoming increasingly popular. Besides, the best theoretically and practically performing gradient-type methods invariably rely on some form of acceleration/momentum to reduce the number of communications (faster convergence), e.g., Nesterov's accelerated gradient descent (Nesterov, 1983, 2004) and Adam (Kingma and Ba, 2014). In order to combine the benefits of communication compression and convergence acceleration, we propose a \emph{compressed and accelerated} gradient method based on ANITA (Li, 2021) for distributed optimization, which we call CANITA. Our CANITA achieves the \emph{first accelerated rate} $O\bigg(\sqrt{\Big(1+\sqrt{\frac{ω^3}{n}}\Big)\frac{L}ε} + ω\big(\frac{1}ε\big)^{\frac{1}{3}}\bigg)$, which improves upon the state-of-the-art non-accelerated rate $O\left((1+\fracω{n})\frac{L}ε + \frac{ω^2+ω}{ω+n}\frac{1}ε ight)$ of DIANA (Khaled et al., 2020) for distributed general convex problems, where $ε$ is the target error, $L$ is the smooth parameter of the objective, $n$ is the number of machines/devices, and $ω$ is the compression parameter (larger $ω$ means more compression can be applied, and no compression implies $ω=0$). Our results show that as long as the number of devices $n$ is large (often true in distributed/federated learning), or the compression $ω$ is not very high, CANITA achieves the faster convergence rate $O\Big(\sqrt{\frac{L}ε}\Big)$, i.e., the number of communication rounds is $O\Big(\sqrt{\frac{L}ε}\Big)$ (vs. $O\big(\frac{L}ε\big)$ achieved by previous works). As a result, CANITA enjoys the advantages of both compression (compressed communication in each round) and acceleration (much fewer communication rounds).

研究动机与目标

• 弥合在分布式凸优化中结合通信压缩与加速的理论差距。
• 设计一种方法，在每轮通信中保持压缩的同时实现更快的收敛速率。
• 将ANITA加速框架扩展至带有压缩算子的分布式设置。
• 在一般凸情形下，建立优于现有方法的通信复杂度的理论保证。

提出的方法

• CANITA通过将ANITA加速框架适配到带有压缩梯度的分布式优化中推导得出。
• 该方法采用由$\omega$参数化的压缩算子，其中$\omega$越大表示压缩程度越高。
• 构建了一种新颖的李雅普诺夫函数以分析收敛性，该函数包含到最优解的距离、梯度追踪以及压缩误差的项。
• 算法使用类似动量的更新方式，并采用自适应步长和动量参数以加速收敛。
• 分析建立了递推不等式，以压缩和加速参数的形式界定向期望次优性。
• 关键技术引理涉及利用凸性和光滑性假设，界定向量在迭代间李雅普诺夫函数的期望下降量。

实验结果

研究问题

• RQ1 通信压缩与收敛加速能否在分布式凸优化中实现理论上的结合？
• RQ2 当同时使用压缩与加速时，可达到的最优收敛速率是什么？
• RQ3 在分布式设置中存在压缩的情况下，加速速率$O(\sqrt{L/\epsilon})$是否仍然成立？
• RQ4 压缩参数$\omega$和设备数量$n$如何影响收敛速率？

主要发现

• CANITA实现了$O\left(\sqrt{\left(1+\sqrt{\frac{\omega^3}{n}}\right)\frac{L}{\epsilon}} + \omega\left(\frac{1}{\epsilon}\right)^{1/3}\right)$的收敛速率，优于非加速的DIANA方法。
• 当$n$较大或$\omega$较小时，CANITA实现了加速速率$O\left(\sqrt{\frac{L}{\epsilon}}\right)$，将通信轮数从$O(L/\epsilon)$减少至$O(\sqrt{L/\epsilon})$。
• 该方法可证明地结合了压缩通信与加速的优势，实现在不牺牲压缩效率的前提下减少通信轮数。
• 分析表明，通过一种新颖的李雅普诺夫函数，压缩引起的方差得到了控制，该函数同时考虑了动量和梯度追踪。
• 在标准假设下实现了收敛速率的改进：目标函数具有$L$-光滑性和凸性。