[Paper Review] An Empirical Study of Federated Learning on IoT-Edge Devices: Resource Allocation and Heterogeneity

Kok‐Seng Wong, Manh Nguyen-Duc+4 more|arXiv (Cornell University)|May 31, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This empirical study evaluates Federated Learning (FL) on real-world IoT-edge devices, revealing that simulation-based FL evaluations significantly underestimate real-world costs and performance issues. The authors demonstrate that computational heterogeneity and data non-IID distribution are the dominant factors affecting FL convergence and efficiency, with on-device training introducing substantial variability in training time and resource consumption across devices.

ABSTRACT

Nowadays, billions of phones, IoT and edge devices around the world generate data continuously, enabling many Machine Learning (ML)-based products and applications. However, due to increasing privacy concerns and regulations, these data tend to reside on devices (clients) instead of being centralized for performing traditional ML model training. Federated Learning (FL) is a distributed approach in which a single server and multiple clients collaboratively build an ML model without moving data away from clients. Whereas existing studies on FL have their own experimental evaluations, most experiments were conducted using a simulation setting or a small-scale testbed. This might limit the understanding of FL implementation in realistic environments. In this empirical study, we systematically conduct extensive experiments on a large network of IoT and edge devices (called IoT-Edge devices) to present FL real-world characteristics, including learning performance and operation (computation and communication) costs. Moreover, we mainly concentrate on heterogeneous scenarios, which is the most challenging issue of FL. By investigating the feasibility of on-device implementation, our study provides valuable insights for researchers and practitioners, promoting the practicality of FL and assisting in improving the current design of real FL systems.

Motivation & Objective

• To investigate the gap between simulated FL evaluations and real-world deployment on IoT-edge devices.
• To analyze the impact of computational heterogeneity, network variability, and data non-IID distribution on FL performance and resource costs.
• To provide empirical insights into on-device FL feasibility, focusing on real resource consumption and system stability.
• To evaluate the limitations of existing simulation-based FL studies in capturing real operational constraints.

Proposed method

• Conducted large-scale, real-world FL experiments across a heterogeneous network of IoT-edge devices, including Raspberry Pi and NVIDIA Jetson platforms.
• Measured actual training time, memory usage, CPU/GPU load, and network bandwidth consumption during each FL round.
• Used the FedAvg algorithm with controlled data partitioning to simulate non-IID data distributions across clients.
• Collected and analyzed system-level metrics (battery drain, memory card performance, server load) to assess real-world operational costs.
• Compared real-device results against simulation-based FL performance to identify discrepancies in convergence speed and resource usage.
• Evaluated the impact of power supply quality and memory card reliability on model training stability and accuracy.

Figure 1: The standard FL framework.

Experimental results

Research questions

• RQ1 RQ1: How do real-world FL deployments on IoT-edge devices differ from simulation-based evaluations in terms of performance and resource consumption?
• RQ2 RQ2: How does computational heterogeneity among IoT devices affect FL convergence time and model accuracy?
• RQ3 RQ3: What is the combined impact of data heterogeneity, network variability, and device resource constraints on FL system efficiency?
• RQ4 RQ4: How do factors like power supply stability and memory card performance influence the reliability and longevity of on-device FL training?

Key findings

• Simulation-based FL evaluations significantly underestimate real-world training time and resource consumption, especially on low-end devices.
• Computational heterogeneity leads to substantial variation in update exchange times, with slower devices becoming bottlenecks in the FL process.
• Data non-IID distribution is the most dominant factor affecting model convergence, especially when combined with computational and network heterogeneity.
• Battery life degradation directly correlates with reduced global model accuracy, indicating that power constraints impact learning quality.
• Memory card speed and reliability affect data I/O performance, with slower or failing cards increasing training latency and risking data loss.
• The aggregation server’s performance is critical; server overload can delay model updates and disrupt the FL training cycle, especially under high client load.

Figure 2: Our Methodology.