[Paper Review] FedNoisy: Federated Noisy Label Learning Benchmark

Siqi Liang, Jintao Huang+4 more|arXiv (Cornell University)|Jun 20, 2023

Machine Learning and Data Classification4 citations

TL;DR

This paper introduces FedNoisy, the first standardized benchmark for federated noisy label learning (FNLL), offering a unified pipeline to simulate 20 diverse FL settings across 5 datasets with varying label noise types, data partitioning schemes, and noise distributions. It reveals that label noise can non-monotonically affect Non-IID settings and increases global gradient norm, providing insights for future FNLL method development.

ABSTRACT

Federated learning has gained popularity for distributed learning without aggregating sensitive data from clients. But meanwhile, the distributed and isolated nature of data isolation may be complicated by data quality, making it more vulnerable to noisy labels. Many efforts exist to defend against the negative impacts of noisy labels in centralized or federated settings. However, there is a lack of a benchmark that comprehensively considers the impact of noisy labels in a wide variety of typical FL settings. In this work, we serve the first standardized benchmark that can help researchers fully explore potential federated noisy settings. Also, we conduct comprehensive experiments to explore the characteristics of these data settings and the comparison across baselines, which may guide method development in the future. We highlight the 20 basic settings for 6 datasets proposed in our benchmark and standardized simulation pipeline for federated noisy label learning, including implementations of 9 baselines. We hope this benchmark can facilitate idea verification in federated learning with noisy labels. exttt{FedNoisy} is available at \codeword{https://github.com/SMILELab-FL/FedNoisy}.

Motivation & Objective

• To address the lack of a standardized benchmark for federated noisy label learning (FNLL) that supports consistent evaluation across diverse settings.
• To simulate realistic federated scenarios involving label noise, data heterogeneity, and varying noise distributions across clients.
• To provide a unified, extensible codebase with ready-to-use FNLL settings and baselines covering multiple CNLL methodology categories.
• To uncover non-monotonic effects of label noise on model performance in Non-IID FL settings.
• To reveal new training dynamics, such as increased gradient norm under noisy labels, that can inform future FNLL method design.

Proposed method

• The benchmark supports 20 basic federated settings combining 5 datasets, multiple data partitioning schemes (IID, Non-IID via Dirichlet, real-world hospital data), and diverse label noise types (symmetric, asymmetric, open-set, globalized, localized).
• A standardized simulation pipeline generates noisy labels with configurable noise ratios and distributions, enabling reproducible FNLL experiments.
• The codebase is built on PyTorch and FedLab, with 8 FNLL baselines integrating state-of-the-art centralized noisy label learning methods into a pluggable FL training workflow.
• Noise is applied using globalized (same across clients) or localized (client-specific) distributions, with noise ratios sampled from uniform distributions for robustness.
• Performance sensitivity to noise ratio is quantified using a gradient-based metric: s(ε) = [acc(ε) - acc(ε+Δ)] / Δ with Δ = 0.1.
• The benchmark enables systematic comparison of FNLL methods under controlled, standardized conditions, including evaluation of gradient norm dynamics during training.

Figure 1: Federated noise scenes provided in FedNoisy . Left: globalized noise ; middle: localized noise ; right: real-word noise . Globalized noise follows a constant noise ratio $\varepsilon_{global}$ in label corruption, while localized noise draws localized noise ratio $\varepsilon_{k}$ from a u

Experimental results

Research questions

• RQ1 How does label noise impact model performance differently in IID versus Non-IID federated learning settings, especially under varying noise ratios?
• RQ2 What is the sensitivity of model accuracy to changes in noise ratio across different data partitioning schemes?
• RQ3 How does label noise affect the global model’s gradient norm during federated training, and does this differ between IID and Non-IID settings?
• RQ4 Can the non-monotonic behavior of performance drop from IID to Non-IID under increasing noise ratios inform new FNLL strategies?
• RQ5 What are the implications of gradient norm amplification due to noisy labels for designing robust FNLL algorithms?

Key findings

• Label noise can cause a non-monotonic effect on performance in Non-IID settings: for certain noise levels, Non-IID may outperform IID, indicating complex interactions between noise and data heterogeneity.
• When noise ratios are low, Non-IID settings are more sensitive to noise, meaning improvements in noise handling yield larger performance gains in such regimes.
• As noise ratios increase, Non-IID becomes less sensitive to further noise increases, while IID becomes more sensitive, suggesting that high-noise systems should prioritize addressing data heterogeneity.
• Noisy labels significantly increase the global model’s gradient norm during training, with IID settings showing greater fluctuation in gradient norm compared to Non-IID.
• The observed increase in gradient norm under noisy labels suggests this metric could serve as a useful signal for detecting and mitigating noisy samples in FNLL.
• The benchmark reveals that current baselines in FNLL are limited in scope, often combining only a few CNLL methods with FL algorithms, highlighting the need for broader methodological evaluation.

Figure 2: Accuracy for different noise ratios on CIFAR-10 with 10 clients. The $x$ -axis noise ratio is $\varepsilon_{global}$ for globalized noise, and $\varepsilon_{local}$ for localized noise $\mathcal{U}(\varepsilon_{local}-0.1,\varepsilon_{local}+0.1)$ . From left to right: globalized noise, lo