[Paper Review] Federated Learning Based on Dynamic Regularization

Durmus Alp Emre Acar, Yue Zhao+4 more|arXiv (Cornell University)|Nov 8, 2021

Privacy-Preserving Technologies in Data36 references114 citations

TL;DR

FedDyn introduces a dynamic regularizer in federated learning to align local device minima with the global objective, achieving better communication efficiency and convergence in convex and non-convex settings across heterogeneous, partially participating, and massively distributed devices.

ABSTRACT

We propose a novel federated learning method for distributively training neural network models, where the server orchestrates cooperation between a subset of randomly chosen devices in each round. We view Federated Learning problem primarily from a communication perspective and allow more device level computations to save transmission costs. We point out a fundamental dilemma, in that the minima of the local-device level empirical loss are inconsistent with those of the global empirical loss. Different from recent prior works, that either attempt inexact minimization or utilize devices for parallelizing gradient computation, we propose a dynamic regularizer for each device at each round, so that in the limit the global and device solutions are aligned. We demonstrate both through empirical results on real and synthetic data as well as analytical results that our scheme leads to efficient training, in both convex and non-convex settings, while being fully agnostic to device heterogeneity and robust to large number of devices, partial participation and unbalanced data.

Motivation & Objective

• Motivate federated learning from a communication-efficiency perspective and address the inconsistency between local and global minima.
• Introduce a dynamic regularizer that guides device updates toward the global stationary point.
• Provide convergence guarantees for convex and non-convex settings under partial participation and device heterogeneity.
• Demonstrate empirical benefits on real and synthetic datasets with reduced communication compared to baselines.

Proposed method

• Propose FedDyn: in each round, a subset of devices optimizes a local penalized objective combining their empirical loss with a dynamic regularizer.
• Dynamic regularizer consists of a linear term based on the device gradient and a quadratic term that couples the device to the server model.
• Device updates satisfy a modified optimality condition that gradually aligns local minima with the global stationary point.
• Server aggregates updated device models and updates its global model to reflect the active participants.
• Provide convergence results showing O(1/T) rates for convex and non-convex settings and linear-like rates in certain strongly convex cases.
• Compare communication costs to FedAvg, FedProx, and SCAFFOLD, highlighting reduced bits transmitted for target accuracy.

Experimental results

Research questions

• RQ1 Can a dynamically updated regularizer ensure that device-level optima converge to stationary points of the global empirical loss?
• RQ2 What convergence rates (convex and non-convex) can be achieved with partial participation and heterogeneous data in FedDyn?
• RQ3 How does FedDyn perform in terms of communication efficiency compared to established FL methods across IID, non-IID, and highly distributed settings?
• RQ4 Is FedDyn robust to device heterogeneity, partial participation, and unbalanced local data while maintaining performance?
• RQ5 Do empirical results on standard benchmarks (MNIST, EMNIST, CIFAR-10/100, Shakespeare) support the theoretical convergence claims?

Key findings

• FedDyn achieves convergence to stationary points of the global loss with a rate of O(1/T) in rounds for convex and non-convex settings.
• For convex smooth local losses with m devices and P active per round, the average loss scales as O(1/T) sqrt(m/P) under balanced data, improving over prior methods like SCAFFOLD.
• In non-convex smooth settings, FedDyn attains an O(1/T) (m/P) rate in expectation for the gradient norm.
• Empirical results on MNIST, EMNIST, CIFAR-10/100, and Shakespeare demonstrate substantial communication savings over FedAvg, FedProx, and SCAFFOLD under various participation and heterogeneity regimes.
• FedDyn requires similar overhead to competing approaches but converges faster, with reduced hyperparameter tuning and robustness to partial participation, large device counts, and data heterogeneity.