[Paper Review] Local Adaptivity in Federated Learning: Convergence and Consistency

Jianyu Wang, Zheng Xu+4 more|arXiv (Cornell University)|Jun 4, 2021

Privacy-Preserving Technologies in Data38 references21 citations

TL;DR

This paper proposes local adaptivity in federated learning by using adaptive optimizers (e.g., AdaGrad) on clients, which accelerates convergence but introduces a non-vanishing solution bias. To resolve this inconsistency, the authors introduce a state-restart mechanism and correction techniques, demonstrating faster convergence and higher test accuracy than baseline methods in real-world federated tasks.

ABSTRACT

The federated learning (FL) framework trains a machine learning model using decentralized data stored at edge client devices by periodically aggregating locally trained models. Popular optimization algorithms of FL use vanilla (stochastic) gradient descent for both local updates at clients and global updates at the aggregating server. Recently, adaptive optimization methods such as AdaGrad have been studied for server updates. However, the effect of using adaptive optimization methods for local updates at clients is not yet understood. We show in both theory and practice that while local adaptive methods can accelerate convergence, they can cause a non-vanishing solution bias, where the final converged solution may be different from the stationary point of the global objective function. We propose correction techniques to overcome this inconsistency and complement the local adaptive methods for FL. Extensive experiments on realistic federated training tasks show that the proposed algorithms can achieve faster convergence and higher test accuracy than the baselines without local adaptivity.

Motivation & Objective

• To investigate the effects of using adaptive optimization methods (e.g., AdaGrad) for local client updates in federated learning.
• To identify that naive use of local adaptive optimizers causes a non-vanishing solution bias, deviating from the global objective's stationary point.
• To propose a state-restart mechanism that resolves discontinuity in optimizer states across communication rounds.
• To develop correction techniques that preserve the benefits of local adaptivity while restoring consistency with the global objective.
• To empirically validate that the proposed method achieves faster convergence and higher test accuracy in realistic federated training tasks.

Proposed method

• Propose a state-restart mechanism: reinitialize client optimizer states (e.g., pre-conditioners, momentum buffers) at the start of each communication round to address discontinuity in stateful optimizers.
• Formulate FedOpt as a fixed-point problem involving an operator $\mathbb{E}[\mathcal{A}]$, where the form of $\mathcal{A}$ depends on the client optimizer, enabling theoretical analysis of convergence and bias.
• Introduce correction techniques that adjust the global update step to counteract the bias introduced by local adaptive methods, preserving convergence to the correct solution.
• Theoretical analysis shows that local adaptivity improves conditioning of the optimization landscape but can shift the fixed point away from the global minimum.
• Use a modified FedOpt framework where ClientOpt uses adaptive methods (e.g., AdaGrad) and ServerOpt uses standard or adaptive updates, with correction applied to the global update step.
• Empirically evaluate the method on realistic federated learning tasks, comparing convergence speed and test accuracy against baselines using only server-side adaptivity or SGD.

Experimental results

Research questions

• RQ1 Can local adaptive optimization methods (e.g., AdaGrad) in federated learning accelerate convergence compared to standard SGD?
• RQ2 Does the use of local adaptive optimizers introduce a non-vanishing solution bias, where the converged model differs from the stationary point of the global objective?
• RQ3 How can the discontinuity in optimizer states across communication rounds be resolved when using stateful local optimizers?
• RQ4 Can correction techniques restore consistency with the global objective while retaining the benefits of local adaptivity?
• RQ5 Does the proposed method achieve faster convergence and higher test accuracy than existing baselines in real-world federated learning scenarios?

Key findings

• Local adaptive methods such as AdaGrad significantly accelerate convergence in federated learning by better conditioning the local optimization landscape.
• Naive use of local adaptive optimizers introduces a non-vanishing solution bias, where the final model converges to a point inconsistent with the global objective's stationary point.
• The proposed state-restart mechanism effectively resolves discontinuity in optimizer states across rounds, enabling stable use of local adaptive methods.
• Correction techniques successfully restore consistency with the global objective while preserving the faster convergence from local adaptivity.
• Experiments on realistic federated tasks show the proposed method achieves faster convergence and higher test accuracy than baselines using only server-side adaptivity or vanilla SGD.
• Theoretical analysis confirms that local adaptivity alters the fixed-point operator $\mathbb{E}[\mathcal{A}]$, and correction is necessary to align the solution with the global minimum.