[Paper Review] Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity

Bo Li, Yasin Esfandiari+3 more|arXiv (Cornell University)|Jun 23, 2023

Privacy-Preserving Technologies in Data54 references4 citations

TL;DR

This paper proposes Fedssyn, a framework that accelerates federated learning convergence under data heterogeneity by shuffling locally generated synthetic data across clients. By theoretically proving that shuffling reduces gradient dissimilarity quadratically and communication rounds super-linearly, Fedssyn achieves performance close to centralized learning without violating data privacy.

ABSTRACT

In federated learning, data heterogeneity is a critical challenge. A straightforward solution is to shuffle the clients' data to homogenize the distribution. However, this may violate data access rights, and how and when shuffling can accelerate the convergence of a federated optimization algorithm is not theoretically well understood. In this paper, we establish a precise and quantifiable correspondence between data heterogeneity and parameters in the convergence rate when a fraction of data is shuffled across clients. We prove that shuffling can quadratically reduce the gradient dissimilarity with respect to the shuffling percentage, accelerating convergence. Inspired by the theory, we propose a practical approach that addresses the data access rights issue by shuffling locally generated synthetic data. The experimental results show that shuffling synthetic data improves the performance of multiple existing federated learning algorithms by a large margin.

Motivation & Objective

• To establish a precise, quantifiable link between data shuffling and convergence rate in federated learning under data heterogeneity.
• To address privacy concerns in data shuffling by replacing real data with locally generated synthetic data.
• To develop a practical, plug-and-play framework compatible with existing federated learning algorithms.
• To empirically validate that synthetic data shuffling improves convergence speed and model accuracy across diverse settings.

Proposed method

• Theoretical analysis proves that shuffling a fraction p of data reduces gradient dissimilarity quadratically and communication rounds super-linearly.
• Proposes Fedssyn: each client trains a local generator on its data to produce synthetic samples, which are then shuffled and redistributed by the server.
• Synthetic data shuffling is applied once at the start, requiring no further coordination or data transfer during FL training.
• The framework is compatible with any FL algorithm and avoids real data sharing, preserving privacy.
• Empirical evaluation uses DDPM and GAN-based generators to produce synthetic data, with performance measured via Top-1 accuracy and communication rounds.
• Theoretical convergence bounds are validated using both strongly convex and non-convex (DNN-based) settings.

Experimental results

Research questions

• RQ1 How does shuffling a fraction of data across clients quantitatively affect the convergence rate of FedAvg under data heterogeneity?
• RQ2 Can synthetic data shuffling replicate the convergence benefits of real data shuffling while preserving data privacy?
• RQ3 What is the impact of synthetic data quality and distribution matching on FL performance?
• RQ4 How does the proportion of shuffled synthetic data influence stochastic noise and gradient dissimilarity in the convergence rate?
• RQ5 Does the theoretical reduction in gradient dissimilarity translate to measurable improvements in communication efficiency and accuracy?

Key findings

• Shuffling a fraction p of data reduces gradient dissimilarity quadratically, leading to a super-linear reduction in required communication rounds—reducing rounds by a factor greater than p.
• At p = 0.06, Fedssyn reduces gradient dissimilarity by 50% and improves Top-1 accuracy by 20% compared to baseline.
• At p = 0.5, the model achieves higher Top-1 accuracy than the IID baseline, demonstrating near-optimal performance under high heterogeneity.
• The effective stochastic noise is dominated by (1−p)σ², indicating that shuffling reduces noise proportionally to the unshuffled portion.
• Empirical results confirm that both stochastic noise and function dissimilarity decrease with synthetic data shuffling, aligning with theoretical predictions.
• DDPM-based generators outperform GAN-based alternatives in generating high-quality synthetic data that improve FL convergence.