[Paper Review] Federated Meta-Learning with Fast Convergence and Efficient Communication

Fei Chen, Mi Luo+3 more|arXiv (Cornell University)|Feb 22, 2018

Privacy-Preserving Technologies in Data36 references277 citations

TL;DR

FedMeta trains a parameterized learning algorithm via federated meta-learning (MAML/Meta-SGD) to enable fast adaptation, reduced communication, and higher accuracy than FedAvg, while preserving client privacy.

ABSTRACT

Statistical and systematic challenges in collaboratively training machine learning models across distributed networks of mobile devices have been the bottlenecks in the real-world application of federated learning. In this work, we show that meta-learning is a natural choice to handle these issues, and propose a federated meta-learning framework FedMeta, where a parameterized algorithm (or meta-learner) is shared, instead of a global model in previous approaches. We conduct an extensive empirical evaluation on LEAF datasets and a real-world production dataset, and demonstrate that FedMeta achieves a reduction in required communication cost by 2.82-4.33 times with faster convergence, and an increase in accuracy by 3.23%-14.84% as compared to Federated Averaging (FedAvg) which is a leading optimization algorithm in federated learning. Moreover, FedMeta preserves user privacy since only the parameterized algorithm is transmitted between mobile devices and central servers, and no raw data is collected onto the servers.

Motivation & Objective

• Motivate federated learning under non-IID and resource-constrained client settings.
• Propose FedMeta, a framework that shares a parameterized algorithm instead of a global model.
• Integrate model-agnostic meta-learning algorithms (MAML and Meta-SGD) into the federated framework.
• Evaluate FedMeta on LEAF datasets and a production dataset to compare against FedAvg in terms of accuracy, convergence, and communication cost.
• Demonstrate privacy benefits by avoiding raw data collection on servers.

Proposed method

• Adopt a parameterized meta-learner ϕ (e.g., initialization θ for MAML or (θ, α) for Meta-SGD) shared by clients.
• Clients receive the algorithm parameters, train on local support data, and report the query loss to update the server-side meta-learner.
• Server updates propagate the meta-learner parameters using aggregated client feedback (outer loop).
• Use episodic meta-training where each episode samples a batch of clients to perform inner/outer updates.
• Integrate MAML and Meta-SGD as running examples within the FedMeta framework to illustrate the approach.
• Demonstrate privacy preservation since only algorithm parameters and losses are transmitted, not raw data.

Experimental results

Research questions

• RQ1 Can FedMeta achieve faster convergence and lower communication costs compared to FedAvg across diverse federated datasets?
• RQ2 Do meta-learning based federated methods (MAML/Meta-SGD) provide higher accuracy and better generalization on non-IID, personalized client data?
• RQ3 How does varying the client support fraction (data used for adaptation) affect FedMeta performance compared to FedAvg?
• RQ4 What are the trade-offs in computation vs communication overhead for FedMeta versus baseline federated learning methods?
• RQ5 Does FedMeta maintain privacy advantages by avoiding raw data collection on the server?

Key findings

• FedMeta reduces required communication cost by 2.82–4.33× across evaluated tasks.
• FedMeta achieves higher final accuracies than FedAvg, with improvements ranging from 3.23% to 14.84%.
• Meta-learning based FedMeta variants (MAML/Meta-SGD) show faster convergence and higher stability on LEAF datasets.
• FedAvg(Meta) often improves over FedAvg, but FedMeta variants typically yield larger gains, especially as the client support fraction increases.
• FedMeta demonstrates effectiveness on a real industrial recommendation task, outperforming both stand-alone and unified federated models in several settings.