[Paper Review] Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies

Yae Jee Cho, Jianyu Wang+1 more|arXiv (Cornell University)|Oct 2, 2020

Privacy-Preserving Technologies in Data221 citations

TL;DR

Introduces biased client selection in federated learning, showing higher local loss focus accelerates convergence, and proposes Power-of-choice with variants to balance speed, bias, and overhead.

ABSTRACT

Federated learning is a distributed optimization paradigm that enables a large number of resource-limited client nodes to cooperatively train a model without data sharing. Several works have analyzed the convergence of federated learning by accounting of data heterogeneity, communication and computation limitations, and partial client participation. However, they assume unbiased client participation, where clients are selected at random or in proportion of their data sizes. In this paper, we present the first convergence analysis of federated optimization for biased client selection strategies, and quantify how the selection bias affects convergence speed. We reveal that biasing client selection towards clients with higher local loss achieves faster error convergence. Using this insight, we propose Power-of-Choice, a communication- and computation-efficient client selection framework that can flexibly span the trade-off between convergence speed and solution bias. Our experiments demonstrate that Power-of-Choice strategies converge up to 3 $\ imes$ faster and give $10$% higher test accuracy than the baseline random selection.

Motivation & Objective

• Motivate and analyze how biased client selection affects FedAvg convergence under data heterogeneity and partial participation.
• Quantify the impact of selecting higher-loss clients on convergence speed.
• Propose the Power-of-Choice client selection framework to trade off convergence speed, bias, and communication/computation costs.
• Develop practical variants to reduce computation and communication overhead without sacrificing gains.
• Validate the approach via experiments across synthetic quadratic problems, synthetic logistic regression, and non-IID FMNIST data.

Proposed method

• Formulates FedAvg with partial client participation and unbiased vs biased client selection.
• Derives convergence analysis under standard smoothness, strong convexity, and bounded gradient assumptions.
• Introduces Local-Global Objective Gap and Selection Skew metrics to capture heterogeneity and bias effects.
• Proposes Power-of-Choice (pi_pow-d) that selects a candidate set of d clients and chooses the highest-loss ones from that set.
• Presents practical variants (pi_cpow-d, pi_rpow-d) to reduce computation and communication, including approximations of local losses.
• Provides experimental evaluation on quadratic optimization, synthetic logistic regression, and non-IID FMNIST.

Experimental results

Research questions

• RQ1 How does biased client selection influence convergence speed and final solution bias in federated averaging with partial participation?
• RQ2 Can selecting high-loss clients accelerate convergence without incurring prohibitive bias or overhead?
• RQ3 What is the trade-off between selection skew (convergence speed) and solution bias under heterogeneity?
• RQ4 Do practical variants of Power-of-Choice maintain gains while reducing communication and computation costs?
• RQ5 How do the proposed strategies perform on synthetic and real-like non-IID federated data?

Key findings

• Biased client selection toward higher local losses can accelerate convergence compared to unbiased/random selection.
• Convergence speed scales with a selection skew parameter, achieving O(1/(T * rho_bar)) under decaying learning rate.
• A non-vanishing bias term arises with biasing, controlled by the ratio rho_tilde/rho_bar; empirical results show this bias can be small.
• Power-of-Choice (pi_pow-d) yields up to ~3x faster convergence and ~10% higher test accuracy than random selection in experiments.
• Increasing the candidate set size d increases convergence speed but can raise final bias; practical variants mitigate costs while preserving gains.
• Variants pi_cpow-d and pi_rpow-d further reduce computation/communication, with competitive or superior accuracy in experiments.
• Experiments on quadratic problems, synthetic federated data, and non-IID FMNIST demonstrate consistent acceleration and accuracy gains over baselines.