[Paper Review] The Non-IID Data Quagmire of Decentralized Machine Learning

Kevin Hsieh, Amar Phanishayee+2 more|arXiv (Cornell University)|Oct 1, 2019

Privacy-Preserving Technologies in Data65 references91 citations

TL;DR

This paper analyzes how skewed label distributions across partitions (non-IID data) degrade decentralized learning performance and proposes SkewScout and Group Normalization as remedies, showing substantial accuracy gains and communication savings.

ABSTRACT

Many large-scale machine learning (ML) applications need to perform decentralized learning over datasets generated at different devices and locations. Such datasets pose a significant challenge to decentralized learning because their different contexts result in significant data distribution skew across devices/locations. In this paper, we take a step toward better understanding this challenge by presenting a detailed experimental study of decentralized DNN training on a common type of data skew: skewed distribution of data labels across devices/locations. Our study shows that: (i) skewed data labels are a fundamental and pervasive problem for decentralized learning, causing significant accuracy loss across many ML applications, DNN models, training datasets, and decentralized learning algorithms; (ii) the problem is particularly challenging for DNN models with batch normalization; and (iii) the degree of data skew is a key determinant of the difficulty of the problem. Based on these findings, we present SkewScout, a system-level approach that adapts the communication frequency of decentralized learning algorithms to the (skew-induced) accuracy loss between data partitions. We also show that group normalization can recover much of the accuracy loss of batch normalization.

Motivation & Objective

• Assess how skewed label partitions affect decentralized learning across multiple models and datasets.
• Evaluate the impact of non-IID data on three representative decentralized learning algorithms.
• Investigate the role of batch normalization versus alternatives under non-IID conditions.
• Propose practical solutions to reduce communication while preserving accuracy in skewed data scenarios.
• Release real-world data and a system-level method to study and mitigate non-IID effects.

Proposed method

• Empirically study decentralized learning with IID and non-IID label partitions across CIFAR-10, ImageNet, LeNet/GoogLeNet/ResNet, and a face recognition setup.
• Evaluate three algorithms: Gaia, FederatedAveraging, and DeepGradientCompression, plus BSP as a baseline.
• Characterize batch normalization divergence under non-IID conditions and test Group Normalization as an alternative.
• Develop SkewScout to estimate data skew and adapt communication by tuning algorithm hyper-parameters to maintain accuracy.
• Use a real-world Flickr-Mammal dataset to illustrate skew in geo-distributed data.
• Provide hyper-parameter sensitivity analyses to show robustness of findings.

Experimental results

Research questions

• RQ1 How does skewed label distribution across data partitions affect model accuracy in decentralized learning?
• RQ2 Do popular decentralized algorithms suffer accuracy loss under non-IID label skew, and is BSP immune?
• RQ3 Is batch normalization particularly vulnerable under non-IID conditions, and can alternatives restore performance?
• RQ4 Can a system-level approach reduce communication while preserving accuracy in skewed data settings?
• RQ5 How effective is a data-skew-aware adaptive communication mechanism (SkewScout) in maintaining model quality?

Key findings

• Skewed label partitions cause major accuracy losses across multiple datasets, models, and decentralized algorithms.
• Batch normalization is especially vulnerable under non-IID label skew, with significant validation accuracy degradation even under BSP.
• The degree of skew strongly determines the difficulty of the problem, with higher skew causing larger accuracy gaps.
• Group Normalization can recover much of the accuracy loss from BatchNorm under non-IID settings, improving performance substantially.
• SkewScout adaptively reduces communication by up to 34.1x (depending on skew) while preserving BSP-level accuracy under non-IID conditions.
• Real-world Flickr-Mammal data demonstrates non-IID degradation is not just synthetic; it occurs in practical geo-distributed scenarios.