[Paper Review] Comparative assessment of federated and centralized machine learning

Ibrahim Abdul Majeed, Sagar Kaushik+5 more|arXiv (Cornell University)|Feb 3, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper compares federated learning (FL) and centralized machine learning, demonstrating that FL offers cost advantages for small to medium models despite challenges from non-IID data distributions. It shows that model size significantly impacts deployment cost in FL, and that careful model design and compression are essential for efficiency, especially when label skewness and limited device data per sample affect convergence.

ABSTRACT

Federated Learning (FL) is a privacy preserving machine learning scheme, where training happens with data federated across devices and not leaving them to sustain user privacy. This is ensured by making the untrained or partially trained models to reach directly the individual devices and getting locally trained "on-device" using the device owned data, and the server aggregating all the partially trained model learnings to update a global model. Although almost all the model learning schemes in the federated learning setup use gradient descent, there are certain characteristic differences brought about by the non-IID nature of the data availability, that affects the training in comparison to the centralized schemes. In this paper, we discuss the various factors that affect the federated learning training, because of the non-IID distributed nature of the data, as well as the inherent differences in the federating learning approach as against the typical centralized gradient descent techniques. We empirically demonstrate the effect of number of samples per device and the distribution of output labels on federated learning. In addition to the privacy advantage we seek through federated learning, we also study if there is a cost advantage while using federated learning frameworks. We show that federated learning does have an advantage in cost when the model sizes to be trained are not reasonably large. All in all, we present the need for careful design of model for both performance and cost.

Motivation & Objective

• To analyze the impact of non-IID data distributions—especially label skew and single-label devices—on federated learning performance.
• To compare federated SGD and federated averaging with centralized gradient descent in terms of convergence and accuracy under skewed data.
• To evaluate the cost implications of federated learning versus centralized training and deployment on cloud platforms.
• To identify key cost drivers in FL, including model size, number of aggregation rounds, and rounds per day.
• To explore model compression and design strategies that reduce model size and deployment cost without sacrificing performance.

Proposed method

• Empirically evaluates federated and centralized training using various data distribution scenarios, including label skew and single-label devices.
• Uses federated averaging and federated SGD with stochastic gradient descent, comparing convergence and accuracy across different data splits.
• Models cost using cloud infrastructure metrics: EC2 instance runtime, S3 data transfer, and message payload size for model updates.
• Analyzes cost as a function of model size, number of aggregation rounds, and rounds per day to isolate key cost drivers.
• Considers model compression techniques such as feature hashing and neural network pruning to reduce model size and deployment cost.
• Evaluates the trade-off between security measures (e.g., differential privacy) and model performance, noting performance degradation with increased anonymization.

Experimental results

Research questions

• RQ1 How does label skew in non-IID data affect the convergence and performance of federated learning compared to centralized learning?
• RQ2 What is the impact of having only one label per device on federated learning performance?
• RQ3 How does model size influence the total cost of federated learning training and deployment on cloud platforms?
• RQ4 What role do the number of aggregation rounds and rounds per day play in determining training and infrastructure costs?
• RQ5 Can model compression and feature reduction techniques effectively reduce deployment cost in federated learning without significant performance loss?

Key findings

• Federated learning with non-IID data, especially single-label or highly skewed label distributions, leads to slower convergence and reduced model accuracy compared to centralized training.
• The number of samples per device and label distribution significantly affect model performance, with too many samples of the same label potentially degrading learning.
• Model deployment cost in federated learning is dominated by S3 reads for model distribution, making model size a critical cost factor.
• Training cost remains relatively stable across different model sizes (e.g., 15KB to 15,000KB), but deployment cost increases sharply with larger models.
• Increasing the number of aggregation rounds per day reduces the total training cost by shortening the required runtime of EC2 instances.
• Model compression techniques such as feature hashing and neural network pruning can reduce model size and lower deployment costs without substantial performance loss.