[Paper Review] Real-World Image Datasets for Federated Learning

Jiahuan Luo, Xueyang Wu+5 more|arXiv (Cornell University)|Oct 14, 2019

Privacy-Preserving Technologies in DataComputer Science23 references67 citations

TL;DR

The paper introduces Street Dataset, a real-world, non-IID image dataset from street cameras for federated learning, and benchmarks YOLOv3 and Faster R-CNN with FedAvg under varied non-IID data partitions.

ABSTRACT

Federated learning is a new machine learning paradigm which allows data parties to build machine learning models collaboratively while keeping their data secure and private. While research efforts on federated learning have been growing tremendously in the past two years, most existing works still depend on pre-existing public datasets and artificial partitions to simulate data federations due to the lack of high-quality labeled data generated from real-world edge applications. Consequently, advances on benchmark and model evaluations for federated learning have been lagging behind. In this paper, we introduce a real-world image dataset. The dataset contains more than 900 images generated from 26 street cameras and 7 object categories annotated with detailed bounding box. The data distribution is non-IID and unbalanced, reflecting the characteristic real-world federated learning scenarios. Based on this dataset, we implemented two mainstream object detection algorithms (YOLO and Faster R-CNN) and provided an extensive benchmark on model performance, efficiency, and communication in a federated learning setting. Both the dataset and algorithms are made publicly available.

Motivation & Objective

• Provide a real-world image dataset reflecting non-IID and unbalanced federated settings.
• Annotate and analyze object distributions across cameras.
• Benchmark two object detectors (YOLOv3 and Faster R-CNN) in a federated learning setup.
• Evaluate model performance, efficiency, and communication costs under federation constraints.

Proposed method

• Collect and annotate street camera images with seven object categories.
• Split data into Street-5 and Street-20 to simulate multi-client federated settings.
• Adopt FedAvg with configurable C (clients per round), E (local epochs), and B (batch size).
• Use PyTorch implementations of YOLOv3 and Faster R-CNN with pretrained backbones (VGG16 for Faster R-CNN, Darknet-53 for YOLOv3).
• Evaluate using IOU and mean average precision (mAP) with 0.5 IOU threshold.
• Provide baseline results and analyze the impact of data non-IIDness and client participation on convergence and communication.

Experimental results

Research questions

• RQ1 How does real-world non-IID and unbalanced data affect federated learning for object detection?
• RQ2 What are the performance and communication trade-offs when applying FedAvg to YOLOv3 and Faster R-CNN on federated Street datasets?
• RQ3 Can pretrained weights mitigate data scarcity and non-IID challenges in federated object detection?
• RQ4 How do different client counts (Street-5 vs Street-20) and local epoch settings influence convergence and mAP?

Key findings

• A real-world Street Dataset with 26 cameras and 7 object categories was created, showing non-IID and unbalanced distributions.
• Two object detectors (YOLOv3 and Faster R-CNN) were benchmarked under FederatedAvg on Street-5 and Street-20 partitions.
• Pretraining significantly improves early convergence and final mAP, especially on Street-20 with fewer samples per client.
• Larger local epochs (E) can speed up early convergence but may not always yield the best final performance in highly non-IID settings.
• FedAvg can achieve competitive performance to centralized training, with convergence behavior depending on C (clients per round) and E (local epochs).
• Communication cost to reach target mAP varies substantially by model size and federation configuration, with YOLOv3 requiring fewer rounds and data transfer than Faster R-CNN.