[Paper Review] Federated Learning via Synthetic Data

Jack Goetz, Ambuj Tewari|arXiv (Cornell University)|Aug 11, 2020

Privacy-Preserving Technologies in Data6 references38 citations

TL;DR

The paper proposes transmitting synthetic data instead of gradient updates in federated learning, achieving large communication reductions with little model degradation.

ABSTRACT

Federated learning allows for the training of a model using data on multiple clients without the clients transmitting that raw data. However the standard method is to transmit model parameters (or updates), which for modern neural networks can be on the scale of millions of parameters, inflicting significant computational costs on the clients. We propose a method for federated learning where instead of transmitting a gradient update back to the server, we instead transmit a small amount of synthetic `data'. We describe the procedure and show some experimental results suggesting this procedure has potential, providing more than an order of magnitude reduction in communication costs with minimal model degradation.

Motivation & Objective

• Motivate reducing client upload communication costs in federated learning.
• Leverage synthetic data to approximate standard gradient updates.
• Develop a practical procedure to generate and transmit synthetic data in FL.
• Enhance the method with improvements that stabilize optimization and evaluate on standard benchmarks.

Proposed method

• Generate synthetic data on clients that induces an update similar to the true local update.
• Use a distillation-based objective to align synthetic-induced updates with actual updates (either via squared error in parameter space or KL-divergence in function space).
• Compute an updateFromSynthetic function that decodes synthetic data into a server update g.
• Aggregate client updates g across the cohort to form the server-side update.
• Employ a backpropagation-based procedure to optimize synthetic data using a Hessian-vector product for efficiency.
• Track a best synthetic data set by evaluating cross-entropy on client data to stabilize optimization.

Experimental results

Research questions

• RQ1 Can synthetic data transmitted from clients approximate the standard gradient updates in federated learning while dramatically reducing communication costs?
• RQ2 How should synthetic data be generated and optimized to best mimic true FL updates, and how robust is the approach to hyperparameters?
• RQ3 What are the trade-offs between communication, compute, and approximation quality in this synthetic-data FL framework?
• RQ4 Can the method be extended to enable server-to-client transmission via synthetic data without prohibitive costs?

Key findings

• Synthetic data FL requires only a small fraction of upload costs (e.g., 2.4% of floats compared to transmitting gradient updates for the MNIST CNN).
• The synthetic-data approach yields model performance comparable to standard full gradient transmission FL on IID and non-IID data partitions.
• The method shows robustness to distillation learning rate across a reasonable range (0.03 to 0.3).
• Increasing synthetic data size or compute improves approximation with diminishing returns, suggesting a balanced mix is optimal.
• Extending the approach to server-to-client transmission is viable but more challenging, achieving substantial download cost reductions with some accuracy trade-offs (around 1.5% loss for over 90% reduction).
• The technique can be stabilized by tracking the best-performing synthetic data using cross-entropy on client data.