[论文解读] FedAvg with Fine Tuning: Local Updates Lead to Representation Learning
在多次本地更新的 FedAvg 在异构客户端之间学习一个共享的低维表示,并在多任务线性回归设置中被严格地证明收敛到真实表示,同时在神经网络中的实证证据表明通过微调实现表示学习。
The Federated Averaging (FedAvg) algorithm, which consists of alternating between a few local stochastic gradient updates at client nodes, followed by a model averaging update at the server, is perhaps the most commonly used method in Federated Learning. Notwithstanding its simplicity, several empirical studies have illustrated that the output model of FedAvg, after a few fine-tuning steps, leads to a model that generalizes well to new unseen tasks. This surprising performance of such a simple method, however, is not fully understood from a theoretical point of view. In this paper, we formally investigate this phenomenon in the multi-task linear representation setting. We show that the reason behind generalizability of the FedAvg's output is its power in learning the common data representation among the clients' tasks, by leveraging the diversity among client data distributions via local updates. We formally establish the iteration complexity required by the clients for proving such result in the setting where the underlying shared representation is a linear map. To the best of our knowledge, this is the first such result for any setting. We also provide empirical evidence demonstrating FedAvg's representation learning ability in federated image classification with heterogeneous data.
研究动机与目标
- Motivate understanding why FedAvg with fine-tuning generalizes well in heterogeneous federated settings.
- Formalize a multi-task linear regression setup with a shared k-dimensional representation.
- Prove that FedAvg with tau >= 2 local updates learns the ground-truth representation at a linear rate.
- Provide empirical evidence of representation learning in neural networks on real datasets.
提出的方法
- Model the problem as multi-task linear regression with a shared d-by-k representation B* and client-specific heads w_i*.
- Analyze FedAvg with tau >= 2 local updates under population losses and equal client weightings.
- Prove convergence of the learned representation dist(B_T, B_*) to within epsilon in T rounds with explicit dependence on diversity and conditioning.
- Show that D-GD (tau=1) cannot recover the ground-truth representation.
- Extend insights to nonlinear settings via experiments on CNNs and evaluate representation preservation during fine-tuning using representation similarity metrics.
实验结果
研究问题
- RQ1Can FedAvg with multiple local updates recover the shared representation across heterogeneous client distributions?
- RQ2Does increasing the number of local updates (tau>=2) enable learning of the common representation that individual client heads can utilize?
- RQ3Is D-GD (tau=1) sufficient to learn the shared representation in this setting?
- RQ4Do empirical experiments with neural networks exhibit representation learning and robustness to fine-tuning on new tasks?
主要发现
- FedAvg with tau>=2 local updates learns the ground-truth representation at a linear rate in the multi-task linear regression setting.
- More diverse local heads across clients enable energy transfer from the ground-truth subspace to the learned representation, driving convergence.
- A one-step local update (D-GD) fails to learn the shared representation regardless of initialization.
- Experiments with neural networks on real datasets show FedAvg and fine-tuning produce representations that change less under fine-tuning compared to Distributed-SGD, indicating representation learning.
- Theoretical results show convergence under population losses and mild initial conditions, with exponential decay of the principal angle distance between learned and ground-truth representations.
阅读本论文
在 Nubint 阅读器中打开原文,边读边就任意段落提问。
在 Nubint 中阅读本论文本解读由 AI 根据论文原文生成,引用前请核对原文。