[论文解读] FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization

Amirhossein Reisizadeh, Aryan Mokhtari+3 more|arXiv (Cornell University)|Sep 28, 2019

Privacy-Preserving Technologies in Data参考文献 40被引用 260

一句话总结

FedPAQ 是一种通信高效的联邦学习算法，结合周期性平均、部分设备参与和量化信息传递，在强凸和非凸损失下具有近最优保证。

ABSTRACT

Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systems-oriented challenges which include (i) communication bottleneck since a large number of devices upload their local updates to a parameter server, and (ii) scalability as the federated network consists of millions of devices. Due to these systems challenges as well as issues related to statistical heterogeneity of data and privacy concerns, designing a provably efficient federated learning method is of significant importance yet it remains challenging. In this paper, we present FedPAQ, a communication-efficient Federated Learning method with Periodic Averaging and Quantization. FedPAQ relies on three key features: (1) periodic averaging where models are updated locally at devices and only periodically averaged at the server; (2) partial device participation where only a fraction of devices participate in each round of the training; and (3) quantized message-passing where the edge nodes quantize their updates before uploading to the parameter server. These features address the communications and scalability challenges in federated learning. We also show that FedPAQ achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by our method.

研究动机与目标

• 解决联邦学习中大量设备的通信瓶颈。
• 在设备可用性部分参与的情况下实现可扩展训练。
• 通过量化通信降低上行数据量，同时保持收敛保证。
• 为强凸和非凸损失函数提供理论保证。

提出的方法

• 引入周期性平均，在本地模型经过设定迭代次数后再进行同步更新。
• 通过每轮随机抽样设备子集来实现部分设备参与。
• 对设备发送到服务器的更新进行量化以降低通信成本。
• 提供一种具体的量化方案（低精度量化器）来对更新进行量化。
• 提出一种聚合量化更新以形成下一个全局模型的算法（FedPAQ）。
• 在适用于强凸和非凸损失的假设下分析收敛性。

实验结果

研究问题

• RQ1 在周期性平均、部分参与和量化的情况下，FedPAQ 能否在不牺牲收敛性的前提下实现通信效率？
• RQ2 在对量化和随机梯度现实假设下，强凸与非凸情形下 FedPAQ 的收敛保证是什么？
• RQ3 周期长度、参与率和量化对通信成本与收敛速率之间的权衡有何影响？

主要发现

• 在适当条件下，FedPAQ 对强凸损失在期望意义上达到 O(1/T) 收敛。
• 对于非凸损失，FedPAQ 达到 O(1/√T) 的收敛速率，达到一阶驻点，周期长度允许达到 O(√T)。
• 理论结果考虑了量化方差、部分参与以及周期性平均中固有的本地更新偏差。
• 该框架显示出明确的通信-计算权衡，量化可以显著降低通信轮数。
• 特殊情况回退到已知结果（例如完全参与、无量化）同时放宽了标准梯度有界性假设。