[论文解读] A Federated Learning Framework for Privacy-preserving and Parallel Training.

Tien-Dung Cao, Tram Truong-Huu+2 more|arXiv (Cornell University)|Jan 22, 2020

Privacy-Preserving Technologies in Data参考文献 23被引用 12

一句话总结

该论文提出FEDF，一种联邦学习框架，可在不暴露原始数据或中间梯度的情况下，实现跨地理分布数据集的隐私保护并行深度学习模型训练。该框架实现最高4.8倍的加速，模型性能与集中式训练相比仅下降4.5%，同时与现有方法相比，通信开销减少高达34%。

ABSTRACT

The deployment of such deep learning in practice has been hurdled by two issues: the computational cost of model training and the privacy issue of training data such as medical or healthcare records. The large size of both learning models and datasets incurs a massive computational cost, requiring efficient approaches to speed up the training phase. While parallel and distributed learning can address the issue of computational overhead, preserving the privacy of training data and intermediate results (e.g., gradients) remains a hard problem. Enabling parallel training of deep learning models on distributed datasets while preserving data privacy is even more complex and challenging. In this paper, we develop and implement FEDF, a distributed deep learning framework for privacy-preserving and parallel training. The framework allows a model to be learned on multiple geographically-distributed training datasets (which may belong to different owners) while do not reveal any information of each dataset as well as the intermediate results. We formally prove the convergence of the learning model when training with the developed framework and its privacy-preserving property. We carry out extensive experiments to evaluate the performance of the framework in terms of speedup ratio, the approximation to the upper-bound performance (when training centrally) and communication overhead between the master and training workers. The results show that the developed framework achieves a speedup of up to 4.8x compared to the centralized training approach and maintaining the performance approximation of the models within 4.5% of the centrally-trained models. The proposed framework also significantly reduces the amount of data exchanged between the master and training workers by up to 34% compared to existing work.

研究动机与目标

• 解决在分布式数据集上训练大型深度学习模型所面临的高计算成本问题。
• 在分布式训练中保护数据隐私，尤其针对医疗记录等敏感数据。
• 在不暴露原始数据或梯度的情况下，实现多个数据拥有者之间的并行训练。
• 确保在所提出的框架下模型能够收敛。
• 最小化主节点与分布式工作节点之间的通信开销。

提出的方法

• 该框架采用联邦学习架构，每个工作节点在其本地数据集上独立训练本地模型，且不共享原始数据。
• 梯度在中央主节点处聚合，且不暴露单个数据或中间值。
• 隐私保护聚合机制确保在模型更新过程中，任何单个工作节点的数据或梯度均不会被暴露。
• 该框架使用安全聚合技术，保护通信过程中训练数据和梯度信息的安全。
• 通过优化动力学的理论分析，形式化证明了在该框架下模型的收敛性。
• 通过减少主节点与工作节点之间交换的数据量，提升了通信效率，与先前方法相比，通信开销最高减少34%。

实验结果

研究问题

• RQ1 一个分布式深度学习框架是否能在保护数据隐私的同时实现显著加速？
• RQ2 联邦模型的性能与集中式训练模型相比有多接近？
• RQ3 该框架在分布式训练中能将通信开销降低到何种程度？
• RQ4 在隐私约束下，所提出的框架是否能保持模型收敛？
• RQ5 该框架在保护医疗记录等敏感数据方面效果如何？

主要发现

• FEDF框架相比集中式训练，最高实现4.8倍加速。
• FEDF的模型性能与集中式训练模型相比，差距在4.5%以内。
• 与现有方法相比，该框架将主节点与工作节点之间的通信开销减少了高达34%。
• 形式化证明确认了在FEDF框架下模型的收敛性。
• 该框架成功保护了训练数据和中间梯度的隐私。
• 该框架在训练医疗记录等敏感分布式数据集的深度学习模型方面具有显著有效性。