[论文解读] Communication-Efficient Federated Learning with Compensated Overlap-FedAvg

Yuhao Zhou, Qing Ye+1 more|arXiv (Cornell University)|Dec 12, 2020

Privacy-Preserving Technologies in Data参考文献 48被引用 5

一句话总结

该论文提出 Overlap-FedAvg，一种通信高效的联邦学习框架，通过将本地模型训练与全局模型通信重叠，消除了空闲等待时间。通过引入分层计算策略、数据补偿机制以及Nesterov加速梯度，该方法在保持模型准确率的同时，显著降低了通信开销并加快了收敛速度，即使在低带宽条件下也能保持高效。

ABSTRACT

Petabytes of data are generated each day by emerging Internet of Things (IoT), but only few of them can be finally collected and used for Machine Learning (ML) purposes due to the apprehension of data & privacy leakage, which seriously retarding ML's growth. To alleviate this problem, Federated learning is proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning introduces massive communication overhead as the synchronized data in each epoch is of the same size as the model, and thereby leading to a low communication efficiency. Consequently, variant methods mainly focusing on the communication rounds reduction and data compression are proposed to reduce the communication overhead of federated learning. In this paper, we propose Overlap-FedAvg, a framework that parallels the model training phase with model uploading & downloading phase, so that the latter phase can be totally covered by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg is further developed with a hierarchical computing strategy, a data compensation mechanism and a nesterov accelerated gradients~(NAG) algorithm. Besides, Overlap-FedAvg is orthogonal to many other compression methods so that they can be applied together to maximize the utilization of the cluster. Furthermore, the theoretical analysis is provided to prove the convergence of the proposed Overlap-FedAvg framework. Extensive experiments on both conventional and recurrent tasks with multiple models and datasets also demonstrate that the proposed Overlap-FedAvg framework substantially boosts the federated learning process.

研究动机与目标

• 解决因客户端与服务器之间频繁传输模型权重而导致的联邦学习中高通信开销问题。
• 通过将本地训练与模型上传/下载阶段重叠，消除通信过程中的空闲等待时间。
• 在重叠训练与通信的同时，保持收敛速度和模型准确率。
• 设计一种隐私保护机制，无需中央服务器知晓客户端的学习率。
• 实现与现有压缩技术的兼容，以进一步提升通信效率。

提出的方法

• Overlap-FedAvg 通过允许客户端在上传/下载模型的同时继续训练，解耦了训练与通信阶段，从而实现了计算与通信的重叠。
• 采用分层计算策略，以管理异步客户端更新，减少模型参数的延迟问题。
• 引入数据补偿机制，通过缩放梯度和由超参数 λ 控制的 Hessian 近似，校正因重叠导致的过时梯度。
• 框架集成了 Nesterov 加速梯度（NAG），以提升收敛速度和模型性能。
• 中央服务器采用固定学习率（例如 1.0）进行补偿，确保隐私性，因为实际客户端学习率对服务器不可见。
• 理论分析证明了在温和假设下的收敛性，且该方法与现有压缩技术正交，可联合使用。

实验结果

研究问题

• RQ1 在联邦学习中，通过重叠训练与通信阶段，能否显著减少空闲时间并提升通信效率？
• RQ2 如何在不泄露客户端特定超参数（如学习率）的前提下，有效补偿因重叠导致的模型延迟？
• RQ3 与原始 FedAvg 相比，所提出的框架是否能保持收敛速度和最终模型准确率？
• RQ4 补偿机制能否设计为具有隐私保护性，特别是在中央服务器不知道客户端学习率的情况下？
• RQ5 Overlap-FedAvg 与其它通信压缩技术结合的潜力有多大？

主要发现

• Overlap-FedAvg 通过重叠训练与通信，减少了通信开销，实现了持续的本地计算，最小化了空闲时间。
• 数据补偿机制有效缓解了过时梯度的负面影响，保持了收敛速度和最终模型准确率。
• 结合 Nesterov 加速梯度后，Overlap-FedAvg 在更少的训练迭代次数和更短的时钟时间下，达到了比 FedAvg 更高的准确率。
• 即使在低带宽条件下，该框架在图像分类和自然语言处理任务中，仍能保持与 FedAvg 几乎相同的最终准确率。
• 理论分析证实了 Overlap-FedAvg 在标准假设下的收敛性，支持其可靠性。
• 该方法与现有压缩技术正交，结合使用可进一步降低通信开销。