[论文解读] Expanding the Reach of Federated Learning by Reducing Client Resource Requirements

Sebastian Caldas, Jakub Konečný+2 more|arXiv (Cornell University)|Dec 18, 2018

Privacy-Preserving Technologies in Data参考文献 24被引用 158

一句话总结

该论文提出两种服务器到客户端通信的压缩策略：基于 Kashin 的有损模型压缩和 Federated Dropout，在不损失准确性的前提下，实现最高可达 14x 的服务器到客户端和 28x 的客户端到服务器通信削减。

ABSTRACT

Communication on heterogeneous edge networks is a fundamental bottleneck in Federated Learning (FL), restricting both model capacity and user participation. To address this issue, we introduce two novel strategies to reduce communication costs: (1) the use of lossy compression on the global model sent server-to-client; and (2) Federated Dropout, which allows users to efficiently train locally on smaller subsets of the global model and also provides a reduction in both client-to-server communication and local computation. We empirically show that these strategies, combined with existing compression approaches for client-to-server communication, collectively provide up to a $14 imes$ reduction in server-to-client communication, a $1.7 imes$ reduction in local computation, and a $28 imes$ reduction in upload communication, all without degrading the quality of the final model. We thus comprehensively reduce FL's impact on client device resources, allowing higher capacity models to be trained, and a more diverse set of users to be reached.

研究动机与目标

• Motivate reducing communication bottlenecks in Federated Learning on heterogeneous edge networks.
• Introduce two strategies to cut server-to-client and local computation costs without sacrificing final model accuracy.

提出的方法

• Apply a basis transform and quantization pipeline (Kashin’s representation) to compress server-to-client model downloads.
• Use Federated Dropout to train sub-models on clients, reducing exchanged model size and local computation.
• Demonstrate compatibility and additive benefits when combined with existing client-to-server compression methods.
• Provide end-to-end workflow showing decompression, local training, and aggregation at the server.

实验结果

研究问题

• RQ1 Can lossy compression of server-to-client model downloads maintain accuracy when the model is decompressed on client devices?
• RQ2 Does Federated Dropout reduce communication and local computation without degrading final model performance?
• RQ3 How do Kashin’s representation and subsampling interact with existing client-to-server compression in FL?
• RQ4 What end-to-end communication and computation savings are achievable when combining these strategies?
• RQ5 What guidance emerges for practical parameter settings (dropout rates, quantization levels) in heterogeneous FL settings?

主要发现

方案	客户端到服务器（传输）	s（客户端到服务器）	q（客户端到服务器）	服务器到客户端（传输）	s（服务器到客户端）	q（服务器到客户端）
激进	Kashin’s	0.4	2	Kashin’s	1.0	3
中等	Kashin’s	0.5	4	Kashin’s	1.0	5
保守	Kashin’s	1.0	8	Kashin’s	1.0	8

• Lossy compression with Kashin’s representation can quantize models down to 4 bits, achieving substantial communication reduction while maintaining accuracy.
• Federated Dropout can match or improve final accuracy with dropout rates around 0.75, while reducing both communication and local computation.
• Combined with existing client-to-server compression, the approach yields up to 14x server-to-client, 28x client-to-server, and 1.7x reduction in local computation without accuracy loss in several datasets.
• In CIFAR-10, server-to-client savings reach ~10x and local computation ~1.3x when using Federated Dropout with compression.
• The combination introduces a slightly slower convergence rate but preserves final model quality across tested benchmarks (MNIST, CIFAR-10, EMNIST).