[Paper Review] Fusion of Global and Local Knowledge for Personalized Federated Learning

Tiansheng Huang, Li Shen+3 more|arXiv (Cornell University)|Feb 21, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes FedSLR, a two-stage proximal algorithm that fuses low-rank global knowledge representation (GKR) and sparse personalized components for personalized federated learning. By decomposing models into compact, low-rank global and sparse local components, FedSLR achieves state-of-the-art performance on personalized tasks, reduces model parameters by 40.11% and downlink communication by 38.34%, while theoretically guaranteeing sub-linear convergence to stationary points.

ABSTRACT

Personalized federated learning, as a variant of federated learning, trains customized models for clients using their heterogeneously distributed data. However, it is still inconclusive about how to design personalized models with better representation of shared global knowledge and personalized pattern. To bridge the gap, we in this paper explore personalized models with low-rank and sparse decomposition. Specifically, we employ proper regularization to extract a low-rank global knowledge representation (GKR), so as to distill global knowledge into a compact representation. Subsequently, we employ a sparse component over the obtained GKR to fuse the personalized pattern into the global knowledge. As a solution, we propose a two-stage proximal-based algorithm named extbf{Fed}erated learning with mixed extbf{S}parse and extbf{L}ow- extbf{R}ank representation (FedSLR) to efficiently search for the mixed models. Theoretically, under proper assumptions, we show that the GKR trained by FedSLR can at least sub-linearly converge to a stationary point of the regularized problem, and that the sparse component being fused can converge to its stationary point under proper settings. Extensive experiments also demonstrate the superior empirical performance of FedSLR. Moreover, FedSLR reduces the number of parameters, and lowers the down-link communication complexity, which are all desirable for federated learning algorithms. Source code is available in \url{https://github.com/huangtiansheng/fedslr}.

Motivation & Objective

• To address the challenge of balancing global knowledge representation and personalized pattern learning in personalized federated learning (PFL) under non-IID data.
• To reduce model and communication complexity in PFL by decomposing personalized models into low-rank and sparse components.
• To design a method that optimally fuses global and personalized knowledge without over-parameterizing local models.
• To theoretically establish convergence guarantees for both the global knowledge representation and the personalized component.
• To empirically validate superior performance, parameter efficiency, and communication reduction in real-world FL settings.

Proposed method

• Proposes a low-rank and sparse decomposition of personalized models, where the global component is a low-rank representation (GKR) and the personalized component is sparse.
• Employs separate regularization on GKR and the sparse component to enforce compactness and expressiveness.
• Introduces a two-stage proximal-based algorithm, FedSLR, to alternately optimize the global and personalized components.
• Uses a proximal update rule with adaptive learning rates to ensure convergence stability.
• Applies matrix factorization to compress both GKR and the mixed model, reducing parameter count.
• Designs a communication-efficient training mechanism that reduces downlink traffic by transmitting only the compressed GKR and sparse updates.

Experimental results

Research questions

• RQ1 Can a low-rank global knowledge representation (GKR) effectively capture shared knowledge across clients in non-IID federated settings?
• RQ2 Can a sparse component efficiently represent personalized patterns without dominating the parameter space?
• RQ3 Does fusing low-rank GKR with sparse personalization lead to better performance than existing PFL methods?
• RQ4 Can the proposed FedSLR algorithm achieve theoretical convergence to stationary points for both global and personalized components?
• RQ5 To what extent can model compression and communication efficiency be improved through this decomposition?

Key findings

• The GKR model achieved 7.23% higher accuracy than FedAvg on CIFAR-100, demonstrating superior global knowledge representation.
• The mixed model in FedSLR achieved 3.52% higher accuracy than Ditto, the previous SOTA, on personalized tasks.
• The GKR model reduced parameters by 50.40% compared to a non-pruned baseline, and the full mixed model reduced parameters by 40.11%.
• Downlink communication was reduced by 38.34% per FL round due to compressed model transmission.
• Theoretical analysis confirms sub-linear convergence of GKR and convergence of the sparse component to their respective stationary points under proper settings.
• Empirical results show that the method maintains high performance while significantly reducing model and communication overhead.