[论文解读] Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search

Chaoyang He, Murali Annavaram+1 more|arXiv (Cornell University)|Apr 18, 2020

Privacy-Preserving Technologies in Data参考文献 10被引用 30

一句话总结

本文提出 FedNAS，一种在非IID边缘客户端之间协作搜索CNN架构以提升准确性的方法，优于像 FedAvg 这样的人工设计基线。它还给出 AutoFL 系统实现及在 CIFAR-10 的非IID数据分布下的实证结果。

ABSTRACT

Federated Learning (FL) has been proved to be an effective learning framework when data cannot be centralized due to privacy, communication costs, and regulatory restrictions. When training deep learning models under an FL setting, people employ the predefined model architecture discovered in the centralized environment. However, this predefined architecture may not be the optimal choice because it may not fit data with non-identical and independent distribution (non-IID). Thus, we advocate automating federated learning (AutoFL) to improve model accuracy and reduce the manual design effort. We specifically study AutoFL via Neural Architecture Search (NAS), which can automate the design process. We propose a Federated NAS (FedNAS) algorithm to help scattered workers collaboratively searching for a better architecture with higher accuracy. We also build a system based on FedNAS. Our experiments on non-IID dataset show that the architecture searched by FedNAS can outperform the manually predefined architecture.

研究动机与目标

• Motivate automated Federated Learning (AutoFL) to address non-IID data distributions and architecture suitability.
• Propose a FedNAS framework that jointly optimizes architecture and weights across distributed clients.
• Demonstrate that FedNAS can find architectures with higher accuracy than manually designed ones on non-IID data.
• Provide an AutoFL system design and empirical evaluation on CIFAR-10 with non-IID distributions.

提出的方法

• Use MiLeNAS as local NAS searcher to optimize weights and architecture on each client.
• Represent architecture via differentiable mixed-operation cells to enable gradient-based search.
• Federated aggregation of weights and architectures by weighted averaging across clients in rounds.
• Integrate FedNAS with an AutoFL system built on FedML and PyTorch for distributed NAS and training.
• Evaluate non-IID data scenarios on CIFAR-10 and compare against FedAvg with a manually designed DenseNet.

实验结果

研究问题

• RQ1 Can a collaborative NAS framework improve model performance in federated settings with non-IID data?
• RQ2 Does FedNAS find CNN architectures that outperform manually designed baselines under heterogeneous data distributions?
• RQ3 What are the efficiency and communication trade-offs of Federated NAS versus traditional FedAvg in practice?

主要发现

• FedNAS achieves higher test accuracy than FedAvg on non-IID CIFAR-10 distributions (average ~4% improvement).
• During the search phase, FedNAS can already outperform FedAvg using a manually designed architecture.
• FedNAS finds architectures with fewer parameters and competitive performance in distributed settings compared to FedAvg.
• The authors report faster search/evaluation cycles when using distributed GPU resources (e.g., 16 clients) versus centralized/manual search.
• An AutoFL system based on FedNAS demonstrates practical deployment potential for federated NAS tasks.