[Paper Review] Entropy-driven Fair and Effective Federated Learning

Lin Wang, Zhichao Wang+3 more|arXiv (Cornell University)|Jan 29, 2023

Privacy-Preserving Technologies in Data5 citations

TL;DR

This paper proposes an entropy-driven federated learning framework that enhances fairness and model effectiveness by dynamically adjusting client participation and gradient aggregation using loss-based attention. By leveraging exponential weighting based on local loss and temperature scaling, the method ensures clients with higher losses contribute more to the global update, improving convergence and reducing performance disparity across heterogeneous clients.

ABSTRACT

Federated Learning (FL) enables collaborative model training across distributed devices while preserving data privacy. Nonetheless, the heterogeneity of edge devices often leads to inconsistent performance of the globally trained models, resulting in unfair outcomes among users. Existing federated fairness algorithms strive to enhance fairness but often fall short in maintaining the overall performance of the global model, typically measured by the average accuracy across all clients. To address this issue, we propose a novel algorithm that leverages entropy-based aggregation combined with model and gradient alignments to simultaneously optimize fairness and global model performance. Our method employs a bi-level optimization framework, where we derive an analytic solution to the aggregation probability in the inner loop, making the optimization process computationally efficient. Additionally, we introduce an innovative alignment update and an adaptive strategy in the outer loop to further balance global model's performance and fairness. Theoretical analysis indicates that our approach guarantees convergence even in non-convex FL settings and demonstrates significant fairness improvements in generalized regression and strongly convex models. Empirically, our approach surpasses state-of-the-art federated fairness algorithms, ensuring consistent performance among clients while improving the overall performance of the global model.

Motivation & Objective

• Address performance disparity in federated learning due to non-IID data and varying client capabilities.
• Improve convergence and model effectiveness in heterogeneous client environments.
• Ensure fair contribution from all clients, especially those with higher local losses.
• Develop a dynamic aggregation mechanism that adapts to client loss distributions in real time.

Proposed method

• Introduce a temperature-scaled exponential weighting scheme to compute client importance based on local loss values.
• Use cosine similarity between client loss vectors and a uniform guidance vector to determine if fairness correction is needed.
• Apply a fair gradient update by blending local gradients with a weighted average of all clients' gradients, where weights are derived from loss-based attention.
• Perform local training with a hybrid update rule that combines local gradients and the fair global gradient to stabilize optimization.
• Aggregate client updates using a loss-aware weighted average, where weights are computed from final local losses after training.
• Dynamically select clients based on loss thresholds and apply fairness-aware aggregation only when significant loss imbalance is detected.

Experimental results

Research questions

• RQ1 How can client contribution be fairly balanced in federated learning under non-IID data and heterogeneous client performance?
• RQ2 What is the impact of loss-based attention on convergence speed and model accuracy in heterogeneous federated settings?
• RQ3 Can dynamic gradient weighting improve fairness without sacrificing training efficiency or model effectiveness?
• RQ4 How does the proposed method compare to standard FedAvg in terms of robustness to client data heterogeneity?
• RQ5 What role does entropy-driven client selection play in maintaining balanced client participation and performance?

Key findings

• The proposed method achieves faster convergence and higher final model accuracy compared to standard FedAvg under non-IID data settings.
• Clients with higher local losses are consistently assigned higher update weights, leading to more balanced performance across clients.
• The use of temperature-scaled exponential weighting improves robustness to client drift and data imbalance.
• The dynamic thresholding mechanism based on cosine similarity effectively identifies when fairness correction is needed.
• The hybrid local update rule reduces optimization instability and improves convergence stability.
• Empirical results show significant improvement in test accuracy on non-IID benchmark datasets, especially in high-heterogeneity scenarios.