[Paper Review] Client Selection in Federated Learning based on Gradients Importance

Ouiame Marnissi, Hajar El Hammouti+1 more|arXiv (Cornell University)|Nov 19, 2021

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes a client selection strategy in federated learning that selects devices based on the L2 norm of their local gradients to improve communication efficiency and convergence speed. By prioritizing clients with the highest gradient norms, the method achieves significantly higher test accuracy—up to 14% better at 150 rounds—compared to random selection, especially under non-iid data conditions, while requiring minimal additional computation beyond gradient norm calculation.

ABSTRACT

Federated learning (FL) enables multiple devices to collaboratively learn a global model without sharing their personal data. In real-world applications, the different parties are likely to have heterogeneous data distribution and limited communication bandwidth. In this paper, we are interested in improving the communication efficiency of FL systems. We investigate and design a device selection strategy based on the importance of the gradient norms. In particular, our approach consists of selecting devices with the highest norms of gradient values at each communication round. We study the convergence and the performance of such a selection technique and compare it to existing ones. We perform several experiments with non-iid set-up. The results show the convergence of our method with a considerable increase of test accuracy comparing to the random selection.

Motivation & Objective

• To address communication inefficiency in federated learning due to limited bandwidth and heterogeneous data distributions.
• To improve convergence speed and model accuracy in non-iid data settings through intelligent client selection.
• To reduce computational overhead by avoiding expensive loss computations while maintaining performance comparable to loss-based selection.
• To provide theoretical convergence guarantees for the proposed selection mechanism.
• To empirically evaluate the impact of the number of selected clients on model performance across multiple datasets.

Proposed method

• The method selects clients at each communication round based on the L2 norm of their local gradient vectors.
• Clients with the highest gradient norms are prioritized for participation, assuming they contribute more to model updates.
• The selection process is lightweight, requiring only gradient norm computation, avoiding additional loss evaluation.
• The approach is integrated into the federated averaging framework, where selected clients send their gradients to the server for aggregation.
• Theoretical analysis supports convergence under the proposed selection rule, particularly in non-iid settings.
• Experiments compare the method against random selection and highest-loss-based selection across MNIST, FMNIST, and CIFAR-10 datasets.

Experimental results

Research questions

• RQ1 Can selecting clients based on gradient norm magnitude improve convergence speed and test accuracy in federated learning?
• RQ2 How does gradient-norm-based selection compare to random and loss-based client selection in terms of performance and computational cost?
• RQ3 What is the optimal number of clients to select for achieving the best model accuracy across different datasets and data heterogeneity levels?
• RQ4 Does the proposed method maintain strong performance with minimal additional computation compared to loss-based selection?
• RQ5 How does data heterogeneity (controlled by β) affect the relative performance of gradient-norm-based selection?

Key findings

• At 150 communication rounds, the gradient-norm-based selection achieved 81.6% test accuracy on MNIST with β=0.3, a 14% improvement over random selection.
• On the same dataset, the method reduced training loss by 8% compared to random selection at iteration 150.
• When data heterogeneity was low (β=5), random selection performed nearly as well as the proposed method, indicating reduced benefit under less skewed data.
• For FMNIST, selecting 15 devices via gradient norm selection achieved 71.6% accuracy at round 150, outperforming other selection counts.
• With 85 selected devices on FMNIST, the performance of gradient-norm-based and loss-based selection strategies was nearly identical, showing comparable effectiveness at high participation rates.
• Selecting 25 devices yielded the best accuracy on both MNIST and CIFAR-10, while 15 devices were optimal for FMNIST, indicating dataset-dependent optimal selection size.