[Paper Review] Federated Learning on Non-IID Data Silos: An Experimental Study

Qinbin Li, Yiqun Diao+2 more|arXiv (Cornell University)|Feb 3, 2021

Privacy-Preserving Technologies in Data81 references82 citations

TL;DR

The paper presents NIID-Bench, a comprehensive benchmark for evaluating federated learning under diverse non-IID data silos, and empirically analyzes four FL algorithms across nine datasets, showing no single method dominates and non-IID distributions significantly affect performance.

ABSTRACT

Due to the increasing privacy concerns and data regulations, training data have been increasingly fragmented, forming distributed databases of multiple "data silos" (e.g., within different organizations and countries). To develop effective machine learning services, there is a must to exploit data from such distributed databases without exchanging the raw data. Recently, federated learning (FL) has been a solution with growing interests, which enables multiple parties to collaboratively train a machine learning model without exchanging their local data. A key and common challenge on distributed databases is the heterogeneity of the data distribution among the parties. The data of different parties are usually non-independently and identically distributed (i.e., non-IID). There have been many FL algorithms to address the learning effectiveness under non-IID data settings. However, there lacks an experimental study on systematically understanding their advantages and disadvantages, as previous studies have very rigid data partitioning strategies among parties, which are hardly representative and thorough. In this paper, to help researchers better understand and study the non-IID data setting in federated learning, we propose comprehensive data partitioning strategies to cover the typical non-IID data cases. Moreover, we conduct extensive experiments to evaluate state-of-the-art FL algorithms. We find that non-IID does bring significant challenges in learning accuracy of FL algorithms, and none of the existing state-of-the-art FL algorithms outperforms others in all cases. Our experiments provide insights for future studies of addressing the challenges in "data silos".

Motivation & Objective

• Identify key challenges of non-IID data in horizontal federated learning across distributed data silos.
• Develop a comprehensive non-IID data benchmark (NIID-Bench) with six partitioning strategies.
• Evaluate state-of-the-art FL algorithms (FedAvg, FedProx, SCAFFOLD, FedNova) under varied non-IID settings.
• Provide insights and a public leaderboard to guide future FL research.

Proposed method

• Introduce six non-IID partitioning strategies that cover label skew, feature skew, and quantity skew.
• Synthesize distributed non-IID datasets by partitioning real-world datasets to enable controlled imbalance properties.
• Implement NIID-Bench with public code and a leaderboard (link provided in paper).
• Experiment with nine datasets (images and tabular) using standard CNN/MLP architectures and SGD optimization.
• Compare four FL algorithms (FedAvg, FedProx, SCAFFOLD, FedNova) across rounds with top-1 accuracy as the metric.
• Analyze how data skew types influence convergence, stability, and final accuracy.

Experimental results

Research questions

• RQ1 How do common FL algorithms perform under a broad set of non-IID data distributions in distributed data silos?
• RQ2 Which non-IID partitioning strategies reveal the strengths and weaknesses of each FL algorithm?
• RQ3 Is there a single algorithm that consistently outperforms others across diverse non-IID scenarios?
• RQ4 How do data skews (label, feature, quantity) impact learning stability and convergence?
• RQ5 Can NIID-Bench inform future directions for robust federated learning on non-IID data silos?

Key findings

• Non-IID data significantly degrades learning accuracy for FL algorithms.
• No single state-of-the-art FL algorithm dominates across all non-IID settings.
• Label distribution skew is generally more challenging than quantity skew.
• Instability in training due to batch normalization and partial sampling is widespread in non-IID FL.
• NIID-Bench reveals varied algorithm performance across different skew types, underscoring the need for a comprehensive benchmark.
• A public leaderboard and codebase are provided to facilitate future evaluations.