[Paper Review] FOCUS: Dealing with Label Quality Disparity in Federated Learning

Yiqiang Chen, Xiaodong Yang+4 more|arXiv (Cornell University)|Jan 29, 2020

Stochastic Gradient Optimization TechniquesComputer Science8 references39 citations

TL;DR

FOCUS detects and downweights clients with noisy labels in federated learning by using a benchmark dataset and a mutual cross-entropy credibility score to perform credibility-weighted aggregation. It outperforms FedAvg in the presence of label noise.

ABSTRACT

Ubiquitous systems with End-Edge-Cloud architecture are increasingly being used in healthcare applications. Federated Learning (FL) is highly useful for such applications, due to silo effect and privacy preserving. Existing FL approaches generally do not account for disparities in the quality of local data labels. However, the clients in ubiquitous systems tend to suffer from label noise due to varying skill-levels, biases or malicious tampering of the annotators. In this paper, we propose Federated Opportunistic Computing for Ubiquitous Systems (FOCUS) to address this challenge. It maintains a small set of benchmark samples on the FL server and quantifies the credibility of the client local data without directly observing them by computing the mutual cross-entropy between performance of the FL model on the local datasets and that of the client local FL model on the benchmark dataset. Then, a credit weighted orchestration is performed to adjust the weight assigned to clients in the FL model based on their credibility values. FOCUS has been experimentally evaluated on both synthetic data and real-world data. The results show that it effectively identifies clients with noisy labels and reduces their impact on the model performance, thereby significantly outperforming existing FL approaches.

Motivation & Objective

• Motivate the need to address label quality disparity in federated learning for ubiquitous End-Edge-Cloud systems, especially in healthcare.
• Propose a privacy-preserving mechanism to measure local data quality without accessing raw data.
• Develop an opportunistic aggregation scheme that weights client updates by credibility scores.
• Evaluate FOCUS on synthetic and real-world healthcare datasets to demonstrate robustness to noisy labels.

Proposed method

• Maintain a small benchmark dataset on the FL server labeled accurately.
• Clients train local models and the server evaluates local models on the benchmark to obtain LS^k.
• Server evaluates the global model on each client’s local data to obtain LL^k.
• Compute credibility C^k from mutual cross-entropy E^k = LS^k + LL^k via C^k = 1 - exp(alpha E^k) / sum_i exp(alpha E^i).
• Aggregate updates using a credibility-weighted FedAvg: M^s_t = sum_k W^k_{t-1} M^k_t with W^k_t = n_k C^k_t / sum_i (n_i C^i_t).
• Ensure convergence by maintaining weights summing to one and leveraging existing FedAvg convergence properties.

Experimental results

Research questions

• RQ1 Can label noise in local datasets degrade federated learning performance under traditional FedAvg aggregation?
• RQ2 Can a credibility-based mutual cross-entropy measure effectively identify noisy-label clients without accessing raw data?
• RQ3 Does credibility-weighted aggregation mitigate the impact of noisy clients while preserving performance on accurate data?
• RQ4 What is the communication cost impact of incorporating benchmark evaluations into federated learning?

Key findings

• FOCUS can identify noisy clients by comparing local and global model performance on a benchmark and client data, reducing their influence in aggregation.
• Under normal (all-correct labels) conditions, FOCUS matches FedAvg performance; under noisy-label conditions, FOCUS outperforms FedAvg by 5.82% accuracy on USC-HAD.
• FOCUS assigns lower weights to noisy clients, as shown by the example where a lower-weight hospital receives reduced credibility.
• FOCUS converges similarly to FedAvg and can converge faster in some scenarios while reducing overfitting to noisy data.
• Experiments on PD-Tremor show FOCUS achieving 7.24% higher accuracy than FedAvg, indicating robustness to label noise in real-world data.