[论文解读] Think Locally, Act Globally: Federated Learning with Local and Global Representations

Paul Pu Liang, Terrance Liu+6 more|arXiv (Cornell University)|Jan 6, 2020

Privacy-Preserving Technologies in Data被引用 238

一句话总结

提出 LG-FedAvg，一种联邦学习框架，联合学习紧凑的局部表示和全局模型，以在处理数据异质性和公平性的同时减少通信。提供理论和在视觉、多模态和移动数据任务上的广泛实验。

ABSTRACT

Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices. As a result, the global model can be smaller since it only operates on local representations, reducing the number of communicated parameters. Theoretically, we provide a generalization analysis which shows that a combination of local and global models reduces both variance in the data as well as variance across device distributions. Empirically, we demonstrate that local models enable communication-efficient training while retaining performance. We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key. Finally, local models handle heterogeneous data from new devices, and learn fair representations that obfuscate protected attributes such as race, age, and gender.

研究动机与目标

• Motivate the need for efficient federated learning on private, distributed data with non-i.i.d. distributions.
• Propose a framework that learns compact local representations alongside a global model to reduce communication.
• Provide theoretical generalization analysis showing benefits of combining local and global models.
• Empirically validate the approach on image, multimodal, and privacy-aware mobile datasets.
• Showcase capabilities in personalization, handling new devices, and learning fair representations.

提出的方法

• Introduce Local Global Federated Averaging (LG-FedAvg) which jointly learns local representations on devices and a global model operating on these representations.
• Define local encoders l_m with parameters θ_m^ℓ and a global model g with parameters θ^g, trained end-to-end via a global loss L_m^g that links local and global components.
• Aggregate updated global parameters across devices using a weighted average by data fraction N_m/N, as in FedAvg.
• Describe local representation learning options (supervised labels y, unsupervised x, or self-supervised z) and an adversarial extension for fairness.
• Provide an end-to-end objective that synchronizes local representations across devices through the global objective.
• Present an α-interpolation between local and global models f_α(x; v̂, û_m) = α f_ℓ(x; û_m) + (1−α) f_g(x; v̂) and derive its generalization behavior.

实验结果

研究问题

• RQ1 Can learning compact local representations on devices reduce global communication while maintaining or improving predictive performance?
• RQ2 How does combining local and global models affect variance due to data and device heterogeneity?
• RQ3 Can LG-FedAvg handle unseen devices and heterogeneous data better than purely local or purely global approaches?
• RQ4 Can local representations be adapted to promote fairness by obfuscating protected attributes?
• RQ5 What is the optimal balance between local and global components for different levels of device and data variance?

主要发现

• LG-FedAvg outperforms purely local or purely global baselines on CIFAR-10 in local testing and matches FedAvg performance on new testing while using about 50% fewer communicated parameters.
• On CIFAR-10 with non-iid splits, LG-FedAvg consistently outperforms FedAvg and a multitask baseline, especially as device variance increases.
• In VQA, LG-FedAvg achieves higher local-test accuracy with substantial reduction in parameters communicated (9.99e10 vs 13.97e10).
• In a real-world mobile mood prediction task, α-splits between local and global models outperform both extremes in accuracy, illustrating effective personalization with shared learning.
• LG-FedAvg reduces catastrophic forgetting when new, rotated MNIST devices appear, outperforming FedAvg and FedProx in online adaptability.
• Adversarial (adv) training with LG-FedAvg can enforce independence from protected attributes, trading off a small drop in classifier accuracy but improving privacy.