[Paper Review] Federated Learning with Personalization Layers

Manoj Ghuhan Arivazhagan, Vinay K. Aggarwal+2 more|arXiv (Cornell University)|Dec 2, 2019

Privacy-Preserving Technologies in DataComputer Science10 references517 citations

TL;DR

FedPer introduces a base + personalization layers architecture for federated learning, using global base layers trained with FedAvg and local personalization layers trained on-device to address statistical heterogeneity, showing improved performance on non-IID CIFAR partitions and Flickr-AES compared to FedAvg.

ABSTRACT

The emerging paradigm of federated learning strives to enable collaborative training of machine learning models on the network edge without centrally aggregating raw data and hence, improving data privacy. This sharply deviates from traditional machine learning and necessitates the design of algorithms robust to various sources of heterogeneity. Specifically, statistical heterogeneity of data across user devices can severely degrade the performance of standard federated averaging for traditional machine learning applications like personalization with deep learning. This paper pro-posesFedPer, a base + personalization layer approach for federated training of deep feedforward neural networks, which can combat the ill-effects of statistical heterogeneity. We demonstrate effectiveness ofFedPerfor non-identical data partitions ofCIFARdatasetsand on a personalized image aesthetics dataset from Flickr.

Motivation & Objective

• Motivate the challenge of statistical heterogeneity in federated learning for personalization tasks.
• Propose a base + personalization layers model to model user-specific information.
• Develop FedPer algorithms that combine FedAvg for base layers with local SGD for personalization layers.
• Demonstrate effectiveness on non-identically partitioned CIFAR datasets and Flickr-AES datasets.

Proposed method

• Define a neural network with shared base layers and client-specific personalization layers.
• Train base layers globally via FedAvg; train personalization layers locally on each device.
• Server aggregates base layer weights with client-weighted averaging; personalization layers stay on-device.
• Use SGD to update both base and personalization layers between global aggregations.
• Evaluate over non-IID data partitions and varied personalization layer configurations.

Experimental results

Research questions

• RQ1 How do personalization layers affect federated learning performance under non-identical data partitions?
• RQ2 What is the impact of the number of personalization layers on accuracy and fairness across clients?
• RQ3 Do base layers still learn meaningful representations when personalization layers capture user-specific aspects?
• RQ4 How does FedPer compare to standard FedAvg on datasets like CIFAR-10/100 and Flickr-AES?
• RQ5 Can FedPer model personalization without relying on centralized data sharing?

Key findings

• FedPer converges faster and achieves higher client-averaged test accuracy than FedAvg under heterogeneous data partitions.
• FedPer reduces cross-client performance variation, improving fairness across users.
• FedAvg performs poorly on the Flickr-AES personalization task, while FedPer captures user preferences via personalization layers.
• Increasing personalization layers generally helps, with effects varying by dataset and architecture.
• Fine-tuning personalization layers between rounds can further improve client performance in some settings (CIFAR-100) but shows mixed results on Flickr-AES.