[论文解读] Ensemble Distillation for Robust Model Fusion in Federated Learning

Tao Lin, Lingjing Kong+2 more|arXiv (Cornell University)|Jun 12, 2020

Privacy-Preserving Technologies in Data参考文献 84被引用 485

一句话总结

本文提出使用集成蒸馏来实现联邦学习中的鲁棒模型融合，解决聚合过程中的鲁棒性问题。

ABSTRACT

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100, ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the server model can be trained much faster, requiring fewer communication rounds than any existing FL technique so far.

研究动机与目标

• 推动联邦学习中的鲁棒聚合。
• 引入集成蒸馏，作为融合客户端模型的方法。
• 研究基于集成的融合在FL中的鲁棒性收益。
• 提供关于所提方法在性能提升（或鲁棒性）方面的实证见解。

提出的方法

• 提出一个集成蒸馏框架，用于在联邦学习中融合来自多个客户端的模型。
• 利用蒸馏技术将本地模型结合成一个鲁棒的全局表示。
• 概述基于集成的聚合的核心步骤（聚合、蒸馏目标和优化）。
• 讨论相较于标准聚合方法在鲁棒性方面的潜在优势。

实验结果

研究问题

• RQ1 集成蒸馏能否提高联邦学习中模型融合的鲁棒性？
• RQ2 基于集成的聚合在性能和鲁棒性方面与传统的联邦平均相比如何？
• RQ3 影响FL融合中集成蒸馏效果的关键因素有哪些？

主要发现

• 未提供自所给摘录。
• 未提供自所给摘录。
• 未提供自所给摘录。