[Paper Review] FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things

Samiul Alam, Tuo Zhang+9 more|arXiv (Cornell University)|Sep 29, 2023

Privacy-Preserving Technologies in Data7 citations

TL;DR

Introduces FedAIoT, an FL benchmark for AIoT with eight IoT-derived datasets and an end-to-end framework, and analyzes data heterogeneity, FL optimizers, client sampling, label noise, and quantization.

ABSTRACT

There is a significant relevance of federated learning (FL) in the realm of Artificial Intelligence of Things (AIoT). However, most existing FL works do not use datasets collected from authentic IoT devices and thus do not capture unique modalities and inherent challenges of IoT data. To fill this critical gap, in this work, we introduce FedAIoT, an FL benchmark for AIoT. FedAIoT includes eight datasets collected from a wide range of IoT devices. These datasets cover unique IoT modalities and target representative applications of AIoT. FedAIoT also includes a unified end-to-end FL framework for AIoT that simplifies benchmarking the performance of the datasets. Our benchmark results shed light on the opportunities and challenges of FL for AIoT. We hope FedAIoT could serve as an invaluable resource to foster advancements in the important field of FL for AIoT. The repository of FedAIoT is maintained at https://github.com/AIoT-MLSys-Lab/FedAIoT.

Motivation & Objective

• Provide eight authentic IoT datasets spanning diverse modalities and applications for federated learning in AIoT.
• Offer a unified end-to-end FL framework that covers non-IID data partitioning, preprocessing, IoT-friendly models, hyperparameters, and IoT-factor emulation.
• Benchmark FL performance across datasets under varying non-IIDness, optimizers, and client sampling ratios to reveal opportunities and challenges.
• Investigate robustness to erroneous labels and the impact of training with quantized precision (FP32 vs FP16) on IoT devices.

Proposed method

• Curate eight IoT-origin datasets with modalities including accelerometer, wireless signals, drone imagery, audio, and smart-home sensors.
• Propose three non-IID data partitioning schemes tailored to classification, regression, and detection tasks.
• Design AIoT-friendly models suitable for resource-constrained devices (e.g., LSTM, ResNet18, YOLOv8n, BiLSTM, MLP).
• Integrate an end-to-end FL framework covering data preprocessing, hyperparameters, and an IoT-factor emulator.
• Incorporate realistic IoT factors such as erroneous label emulation via a Centralized-trained confusion matrix and support FP16 quantized training for memory efficiency.

Figure 1 : Overview of FedAIoT .

Experimental results

Research questions

• RQ1 How does data heterogeneity (non-IIDness) affect FL performance across diverse AIoT tasks and modalities?
• RQ2 How do standard FL optimizers (FedAvg, FedOPT) compare under varying heterogeneity, and how does client sampling ratio influence accuracy and convergence?
• RQ3 What is the impact of erroneous labels and quantized training on FL performance for AIoT datasets?
• RQ4 Which IoT modalities and tasks benefit most from quantization and resilient training strategies?

Key findings

• FedAvg generally yields more stable performance under low heterogeneity and often approaches centralized results, while FedOPT can diverge under high heterogeneity on several datasets.
• Higher client sampling ratios tend to improve final accuracy but do not always speed up convergence, with modality-dependent effects.
• Erroneous labels significantly degrade performance on several datasets, notably UT-HAR and EPIC-SOUNDS, indicating a need for label-noise robust FL methods.
• Quantized training (FP16) reduces memory usage substantially (60%+), with mixed effects on accuracy across datasets; some datasets maintain or exceed FP32 performance, others suffer.
• The benchmark highlights dataset- and modality-specific resilience to heterogeneity, noise, and compression, underscoring the need for tailored FL strategies in AIoT.

Figure 2 : Architecture of the end-to-end FL framework for AIoT incorporated in FedAIoT .