[Paper Review] End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things

Yansong Gao, Minki Kim+7 more|arXiv (Cornell University)|Mar 30, 2020

Privacy-Preserving Technologies in Data28 references39 citations

TL;DR

The paper empirically compares Federated Learning (FL) and Split Neural Networks (SplitNN) on real IoT hardware, evaluating learning performance under IID, imbalanced, and non-IID data, plus implementation overhead on Raspberry Pi devices.

ABSTRACT

This work is the first attempt to evaluate and compare felderated learning (FL) and split neural networks (SplitNN) in real-world IoT settings in terms of learning performance and device implementation overhead. We consider a variety of datasets, different model architectures, multiple clients, and various performance metrics. For learning performance, which is specified by the model accuracy and convergence speed metrics, we empirically evaluate both FL and SplitNN under different types of data distributions such as imbalanced and non-independent and identically distributed (non-IID) data. We show that the learning performance of SplitNN is better than FL under an imbalanced data distribution, but worse than FL under an extreme non-IID data distribution. For implementation overhead, we end-to-end mount both FL and SplitNN on Raspberry Pis, and comprehensively evaluate overheads including training time, communication overhead under the real LAN setting, power consumption and memory usage. Our key observations are that under IoT scenario where the communication traffic is the main concern, the FL appears to perform better over SplitNN because FL has the significantly lower communication overhead compared with SplitNN, which empirically corroborate previous statistical analysis. In addition, we reveal several unrecognized limitations about SplitNN, forming the basis for future research.

Motivation & Objective

• Measure learning performance (accuracy and convergence) of FL and SplitNN under IID, imbalanced, and non-IID data settings.
• Evaluate end-to-end implementation overhead (training time, power, memory, communication) on resource-constrained IoT hardware.
• Explore the feasibility of deploying FL and SplitNN on Raspberry Pi and derive practical deployment guidance.

Proposed method

• Compare FL and SplitNN across multiple datasets and client counts (2 to 100) under IID, imbalanced, and non-IID conditions.
• Use 1D CNN models with small parameter counts suitable for IoT devices; assess convergence speed and accuracy.
• Mount both FL and SplitNN on Raspberry Pi devices to measure training time, memory, power, temperature, and communication overhead.
• Investigate SplitNN-enabled ensemble learning to generate multiple models in parallel on server resources.
• Extend SplitNN to ensemble learning and compare end-to-end timing and resource usage against training models individually.

Experimental results

Research questions

• RQ1 RQ1: What factors (number of clients, imbalanced data, non-IID data) affect SplitNN learning performance?
• RQ2 RQ2: Under which settings does SplitNN outperform Federated Learning in terms of learning performance and convergence?
• RQ3 RQ3: What is the end-to-end implementation overhead of FL vs SplitNN on resource-constrained IoT hardware?
• RQ4 RQ4: Can SplitNN be effectively used with ensemble learning to reduce training time and resource usage?

Key findings

• SplitNN converges faster than FL with imbalanced data distributions but fails to learn under extreme non-IID distributions.
• FL struggles to reach centralized baselines with many clients or highly non-IID data, though local-epoch tuning can help convergence.
• On Raspberry Pi hardware, FL generally has lower communication overhead and higher client-side parallelism, making it more suitable when communication is the main bottleneck.
• SplitNN incurs higher communication overhead and can be infeasible for heavy models on IoT devices; smaller 1D CNNs are recommended for such hardware.
• Ensemble learning with SplitNN can reduce total training time and some resource usage compared to training models sequentially, though not reducing communication overhead.