[Paper Review] Adaptive Federated Learning in Resource Constrained Edge Computing Systems

Shiqiang Wang, Tiffany Tuor+5 more|arXiv (Cornell University)|Apr 14, 2018

Privacy-Preserving Technologies in DataComputer Science42 references85 citations

TL;DR

The paper analyzes gradient-descent based federated learning on edge nodes with non-i.i.d. data and resource constraints, derives a convergence bound, and proposes a real-time adaptive algorithm to choose global aggregation frequency to minimize loss under resource budgets.

ABSTRACT

Emerging technologies and applications including Internet of Things (IoT), social networking, and crowd-sourcing generate large amounts of data at the network edge. Machine learning models are often built from the collected data, to enable the detection, classification, and prediction of future events. Due to bandwidth, storage, and privacy concerns, it is often impractical to send all the data to a centralized location. In this paper, we consider the problem of learning model parameters from data distributed across multiple edge nodes, without sending raw data to a centralized place. Our focus is on a generic class of machine learning models that are trained using gradient-descent based approaches. We analyze the convergence bound of distributed gradient descent from a theoretical point of view, based on which we propose a control algorithm that determines the best trade-off between local update and global parameter aggregation to minimize the loss function under a given resource budget. The performance of the proposed algorithm is evaluated via extensive experiments with real datasets, both on a networked prototype system and in a larger-scale simulated environment. The experimentation results show that our proposed approach performs near to the optimum with various machine learning models and different data distributions.

Motivation & Objective

• Motivate learning model parameters from distributed edge data without sending raw data to a central location.
• Derive a theoretical convergence bound for gradient-descent based federated learning with non-i.i.d. data and multiple local updates between aggregations.
• Develop a control algorithm that adapts aggregation frequency in real time based on system dynamics and model characteristics.
• Evaluate the proposed approach on real datasets with hardware prototype and simulations to show near-optimal performance.

Proposed method

• Formulate a gradient-descent based federated learning framework with local updates and periodic global aggregation.
• Derive a convergence bound that incorporates non-i.i.d. data distributions and multiple local updates between aggregations (tau).
• Define a resource-aware optimization problem to minimize final loss under multi-resource budgets (M types).
• Propose a control algorithm that learns data distribution, system dynamics, and model characteristics to adapt tau and total iterations T.
• Use theoretical results to approximate solutions to the resource-constrained optimization problem, followed by empirical validation on real datasets and prototypes.

Experimental results

Research questions

• RQ1 How does non-i.i.d. data across edge nodes affect convergence of distributed gradient descent in federated learning?
• RQ2 What is the optimal frequency of global aggregation under given resource budgets to minimize training loss?
• RQ3 Can a real-time control algorithm adapt aggregation frequency to approach near-optimal learning performance across varying models and data distributions?
• RQ4 How do local update steps between aggregations impact overall convergence and resource utilization?

Key findings

• The paper provides a novel convergence bound for distributed gradient descent with non-i.i.d. data and arbitrary numbers of local updates between aggregations.
• A control algorithm is proposed to adapt the global aggregation frequency in real time to minimize learning loss under fixed resource budgets.
• Extensive experiments on real datasets show the approach achieves near-optimal performance across different models, distributions, and system configurations.
• The framework accounts for multiple resource types (e.g., time, energy, bandwidth) and their per-update costs to optimize training efficiency.