[Paper Review] Decentralized federated learning of deep neural networks on non-iid data

Noa Onoszko, Gustav Karlsson+2 more|arXiv (Cornell University)|Jul 18, 2021

Privacy-Preserving Technologies in DataComputer Science24 references27 citations

TL;DR

The paper introduces Performance-Based Neighbor Selection (PENS), a fully decentralized gossip-based federated learning method that identifies peers with similar data distributions to improve learning on non-iid data, showing improvements over baselines and approaching an oracle.

ABSTRACT

We tackle the non-convex problem of learning a personalized deep learning model in a decentralized setting. More specifically, we study decentralized federated learning, a peer-to-peer setting where data is distributed among many clients and where there is no central server to orchestrate the training. In real world scenarios, the data distributions are often heterogeneous between clients. Therefore, in this work we study the problem of how to efficiently learn a model in a peer-to-peer system with non-iid client data. We propose a method named Performance-Based Neighbor Selection (PENS) where clients with similar data distributions detect each other and cooperate by evaluating their training losses on each other's data to learn a model suitable for the local data distribution. Our experiments on benchmark datasets show that our proposed method is able to achieve higher accuracies as compared to strong baselines.

Motivation & Objective

• Address learning personalized deep models in a fully decentralized federated setting with non-iid client data.
• Develop a decentralized algorithm to identify and leverage peers with similar data distributions.
• Evaluate performance of the proposed method against baselines on non-iid data benchmarks.

Proposed method

• Use gossip protocol for peer-to-peer communication without a central server.
• Introduce Pens which first identifies neighbors with similar data distributions by evaluating the loss of each client’s model on others’ data over T rounds.
• Select top-performing neighbors based on lowest losses and merge their models before local training.
• Then continue decentralized gossip with the chosen neighbor set to refine models.
• Provide baseline comparisons (Random Gossip, Local) and an Oracle with perfect distribution information.
• Experiment with CNN on CIFAR-10 and Fashion-MNIST under covariate shift via image rotations.

Experimental results

Research questions

• RQ1 Can a fully decentralized gossip-based approach achieve high accuracy on non-iid data without a central server?
• RQ2 Does performance-based neighbor selection improve learning over random neighbor communication in decentralized FL?
• RQ3 How close can Pens approach an oracle that has perfect knowledge of data distributions across clients?
• RQ4 What are the robustness and parameter sensitivities (n_sampled, m) of Pens under covariate shift?

Key findings

• Pens outperforms random gossip and locally trained baselines on CIFAR-10 and Fashion-MNIST under non-iid rotations.
• With enough local data, Pens achieves accuracies close to an Oracle with perfect distribution knowledge.
• Pens shows robustness to variations in n_sampled and m, though the ratio n_sampled/m should be balanced to avoid collapse to a small peer subset.
• On CIFAR-10 with two or four rotated distributions, Pens improves accuracy over baselines.
• On Fashion-MNIST, Pens consistently surpasses baselines for 100 and 500 training samples per client.
• The approach works under non-iid covariate shift where D_i(x) differ but D_i(y|x) are the same.