[Paper Review] Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization

Jianyu Wang, Qinghua Liu+3 more|arXiv (Cornell University)|Jul 15, 2020

Privacy-Preserving Technologies in Data51 references572 citations

TL;DR

The paper analyzes objective inconsistency caused by heterogeneous local updates in federated optimization, introduces a general framework, and proposes FedNova to normalize updates and achieve consistent convergence.

ABSTRACT

In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.

Motivation & Objective

• Motivate and formalize the objective inconsistency problem arising from unequal local updates in federated learning.
• Develop a general convergence framework that subsumes FedAvg and FedProx under heterogeneous local progress.
• Characterize the bias and slowdown caused by inconsistency and establish theoretical guarantees.
• Propose a normalization-based aggregation method (FedNova) that achieves objective consistency while preserving fast convergence.

Proposed method

• Model updates can be written as a generalized update rule that optimizes a surrogate objective  = sum wi Fi with normalized gradients di(t).
• Define d_i(t) as the normalized gradient using local gradient accumulations a_i and the l1 norm; show how a_i encodes the local solver and schedule.
• Introduce weights w_i and an effective step count tau_eff to capture aggregation and progress across heterogeneous clients.
• Prove convergence to a stationary point of the surrogate objective and bound the gradient norm with respect to both the surrogate and true objectives.
• Show how FedNova sets wi = pi (data share) to eliminate objective inconsistency and derive linear speedups and convergence guarantees.
• Demonstrate compatibility with various local solvers and potential combination with acceleration/variance-reduction techniques.

Experimental results

Research questions

• RQ1 How does heterogeneity in the number of local updates across clients affect convergence in federated optimization?
• RQ2 Can we quantify and mitigate the bias introduced by objective inconsistency in standard aggregation methods like FedAvg and FedProx?
• RQ3 Does a normalization-based aggregation (FedNova) restore consistency without sacrificing convergence speed?
• RQ4 How do the theoretical guarantees extend to non-IID data, varying client participation, and different local solvers?
• RQ5 What are the trade-offs between objective consistency and convergence slowdown when adjusting proximal terms or other hyperparameters?

Key findings

• FedAvg converges to a stationary point of a surrogate objective, which can differ arbitrarily from the true global objective depending on local update counts.
• Objective inconsistency introduces a non-vanishing error floor unless aggregation weights reflect true data shares (pi).
• FedProx can reduce but not eliminate inconsistency and may slow convergence as proximal strength grows.
• A generalized update rule captures many federated algorithms and quantifies the bias and slowdown due to heterogeneous local updates.
• FedNova eliminates objective inconsistency by normalizing and averaging local gradients, achieving consistency and preserving fast error convergence.
• FedNova is compatible with arbitrary local solvers and can be combined with variance reduction and server-side acceleration.