[Paper Review] A Simple Data Augmentation for Feature Distribution Skewed Federated Learning

Yunlu Yan, Fu, Huazhu+5 more|arXiv (Cornell University)|Jun 14, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes FedRDN, a simple yet effective data augmentation method for federated learning under feature distribution skew. By randomly injecting global statistical features (mean and std) from multiple clients into local data during augmentation, FedRDN enhances feature generalization and mitigates domain shift, significantly improving performance across multiple FL benchmarks with minimal overhead.

ABSTRACT

Federated Learning (FL) facilitates collaborative learning among multiple clients in a distributed manner and ensures the security of privacy. However, its performance inevitably degrades with non-Independent and Identically Distributed (non-IID) data. In this paper, we focus on the feature distribution skewed FL scenario, a common non-IID situation in real-world applications where data from different clients exhibit varying underlying distributions. This variation leads to feature shift, which is a key issue of this scenario. While previous works have made notable progress, few pay attention to the data itself, i.e., the root of this issue. The primary goal of this paper is to mitigate feature shift from the perspective of data. To this end, we propose a simple yet remarkably effective input-level data augmentation method, namely FedRDN, which randomly injects the statistical information of the local distribution from the entire federation into the client's data. This is beneficial to improve the generalization of local feature representations, thereby mitigating feature shift. Moreover, our FedRDN is a plug-and-play component, which can be seamlessly integrated into the data augmentation flow with only a few lines of code. Extensive experiments on several datasets show that the performance of various representative FL methods can be further improved by integrating our FedRDN, demonstrating its effectiveness, strong compatibility and generalizability. Code will be released.

Motivation & Objective

• To address feature distribution skew in federated learning, a common yet underexplored challenge in real-world FL deployments.
• To investigate whether input-level data augmentation can effectively mitigate feature shift caused by skewed local data distributions.
• To develop a general, non-parametric, and privacy-preserving data augmentation technique that integrates seamlessly into existing FL pipelines.
• To improve the generalization and robustness of local models across heterogeneous clients without modifying the model architecture.
• To demonstrate scalability and effectiveness across diverse datasets and FL frameworks.

Proposed method

• FedRDN introduces a novel data augmentation strategy that randomly samples global statistical features (mean and standard deviation) from all clients’ local datasets.
• During training, each client applies a modified normalization using a randomly selected global statistic pair (μ, σ) from the federation, rather than using its own local statistics.
• This process is implemented as a plug-and-play component in the data augmentation pipeline, requiring only a few lines of code to integrate.
• The method is non-parametric and incurs negligible computational and communication overhead, as it only exchanges statistical summaries across clients.
• The augmentation is applied at the input level before model forward pass, enabling effective distribution shift mitigation through richer data diversity.
• FedRDN uses a random sampling strategy over global statistics, ensuring that each client sees a wider range of underlying data distributions during training.

Experimental results

Research questions

• RQ1 Can input-level data augmentation effectively mitigate feature distribution skew in federated learning?
• RQ2 Can a simple, non-parametric data augmentation method improve model generalization across clients with skewed data distributions?
• RQ3 How does injecting global statistical features into local data affect model convergence and performance in FL?
• RQ4 Is FedRDN robust to varying local training epochs and data heterogeneity levels?
• RQ5 Does FedRDN outperform existing methods like FedMix and FedFA in addressing feature shift under real-world data skew?

Key findings

• FedRDN significantly improves performance on the Office-Caltech-10 dataset, achieving 87.50% accuracy on the DSLR→Amazon cross-site transfer, outperforming FedAvg by 18.75 percentage points.
• On DomainNet, FedRDN boosts FedAvg accuracy from 42.32% to 43.55%, demonstrating consistent improvement across diverse domains.
• In the ProstateMRI benchmark, FedRDN increases accuracy from 90.02% to 92.32%, showing strong generalization on medical imaging tasks.
• FedRDN achieves better cross-site generalization than FedAvg, FedMix, and FedFA, as shown by consistent performance gains across all client-target combinations.
• The ablation study confirms that FedRDN-V, which uses fixed global statistics instead of random sampling, underperforms FedRDN by up to 8.3 percentage points, proving the importance of stochastic augmentation.
• T-SNE visualization confirms that FedRDN leads to more consistent and less biased feature distributions across clients compared to FedAvg.