[Paper Review] FedMix: Approximation of Mixup under Mean Augmented Federated Learning

Tehrim Yoon, Sumin Shin+2 more|arXiv (Cornell University)|Jul 1, 2021

Privacy-Preserving Technologies in Data34 references67 citations

TL;DR

FedMix proposes an approximation of global Mixup within a Mean Augmented Federated Learning (MAFL) framework, enabling Mixup-like data augmentation using only averaged data to improve non-iid federated learning performance while preserving privacy.

ABSTRACT

Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device, thus preserving privacy and eliminating the need to store data globally. While there are promising results under the assumption of independent and identically distributed (iid) local data, current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases. To resolve this issue, we propose a simple framework, Mean Augmented Federated Learning (MAFL), where clients send and receive averaged local data, subject to the privacy requirements of target applications. Under our framework, we propose a new augmentation algorithm, named FedMix, which is inspired by a phenomenal yet simple data augmentation method, Mixup, but does not require local raw data to be directly shared among devices. Our method shows greatly improved performance in the standard benchmark datasets of FL, under highly non-iid federated settings, compared to conventional algorithms.

Motivation & Objective

• Motivate federated learning under data heterogeneity (non-iid) and privacy constraints.
• Introduce MAFL as a framework that exchanges averaged data alongside model parameters.
• Develop FedMix to approximate global Mixup using averaged data via a Taylor expansion.
• Show empirical gains over FedAvg and FedProx on standard FL benchmarks, especially in non-iid settings.

Proposed method

• Define MAFL where clients exchange averaged data in addition to model parameters.
• Propose NaiveMix as a baselineMixup between local data and averaged external data.
• Develop FedMix by approximating the global Mixup loss with a Taylor expansion around lambda << 1, using only averaged data.
• Show that the FedMix loss reduces to an expression involving local data and averaged external means, plus a gradient term with external means.
• Evaluate on FEMNIST, CIFAR-10, CIFAR-100, and Shakespeare to compare against FedAvg, FedProx, and Local/NaiveMix variants.

Experimental results

Research questions

• RQ1 Can averaging data across clients (MAFL) improve model performance under non-iid data without sharing raw data?
• RQ2 How effectively can FedMix approximate the effect of global Mixup using only averaged data?
• RQ3 What is the impact of MAFL parameters (e.g., M_k, lambda) on privacy, communication, and accuracy?
• RQ4 Does FedMix consistently outperform baseline FL methods across diverse non-iid scenarios and datasets?

Key findings

• FedMix achieves higher accuracy and faster convergence than FedAvg, FedProx, LocalMix, and NaiveMix on FEMNIST, CIFAR-10, and CIFAR-100 in non-iid settings.
• FedMix shows strong robustness to increasing heterogeneity, including scenarios with fewer clients per round.
• In Shakespeare (language data), MAFL variants improve over baselines, though global Mixup is not consistently superior.
• Varying M_k (mean data size) has limited impact on performance, suggesting effective privacy-preserving averaging.
• FedMix outperforms even when compared to FedProx variants combined with Mixup strategies, across multiple experiments.
• The approach maintains privacy better than global Mixup while achieving competitive or superior accuracy.