[Paper Review] Peer-to-peer Federated Learning on Graphs

Anusha Lalitha, Osman Cihan Kilinc+2 more|arXiv (Cornell University)|Jan 31, 2019

Privacy-Preserving Technologies in Data11 references139 citations

TL;DR

The paper introduces a fully decentralized peer-to-peer federated learning algorithm on graphs where nodes update Bayesian-like beliefs using their local data and one-hop neighbors, proving high-probability guarantees and validating on linear regression and DNN with variational inference.

ABSTRACT

We consider the problem of training a machine learning model over a network of nodes in a fully decentralized framework. The nodes take a Bayesian-like approach via the introduction of a belief over the model parameter space. We propose a distributed learning algorithm in which nodes update their belief by aggregate information from their one-hop neighbors to learn a model that best fits the observations over the entire network. In addition, we also obtain sufficient conditions to ensure that the probability of error is small for every node in the network. We discuss approximations required for applying this algorithm to train Deep Neural Networks (DNNs). Experiments on training linear regression model and on training a DNN show that the proposed learning rule algorithm provides a significant improvement in the accuracy compared to the case where nodes learn without cooperation.

Motivation & Objective

• Motivate learning without a central server by leveraging networked nodes with localized data.
• Develop a Bayesian-like belief update plus neighbor consensus to learn a global model.
• Provide high-probability guarantees on achieving the globally optimal model across all nodes.
• Specialize the framework to linear regression and deep neural networks.
• Demonstrate practical feasibility with variational inference for scalable DNN training.

Proposed method

• Model the network as a directed graph with a stochastic weight matrix W capturing neighbor influence.
• Each node maintains a private belief and a public belief over a finite parameter set Theta and performs a local Bayesian update using its data.
• Nodes exchange public beliefs with one-hop neighbors and update their private belief via a log-belief consensus step.
• Declare the estimate hat{theta} as the argmax of the current private belief at each iteration.
• Provide theoretical guarantees showing high probability that all nodes converge to Theta* under mild connectivity and learnability assumptions.
• When applying to DNNs, replace exact Bayesian updates with variational inference to obtain tractable approximations.

Experimental results

Research questions

• RQ1 How can learning be performed in a fully decentralized network without a central server?
• RQ2 What are the sample complexity and network connectivity requirements to ensure all nodes learn a globally optimal parameter?
• RQ3 Can the framework learn accurately for both linear and nonlinear (DNN) models?
• RQ4 How can variational inference be employed to scale Bayesian updates for complex models?
• RQ5 What performance gains are achievable compared to isolated local learning or centralized training?

Key findings

• Theorem 1 provides a high-probability bound: P(exists i with hat{theta}^{(n)}_i not in Theta*) ≤ delta when n ≥ 16 C log(NM/δ) / (K(Theta)^2 (1 − λ_max(W))).
• Corollary 1 gives an upper bound on the network-averaged true risk difference, scaling with sqrt(r) under Assumption 4.
• The algorithm achieves learning of the globally optimal model under mild connectivity and learnability, with the lower bound on training samples growing logarithmically with the number of nodes and parameters.
• Experiments on distributed Bayesian linear regression show decentralized learning matching central-server performance when cooperation is used.
• For DNNs, VI-based updates enable scalable decentralized training with negligible accuracy loss compared to centralized baselines in provided setups.
• Empirical results on distributed MNIST fashion data demonstrate accurate distributed training under both IID and non-IID, balanced and unbalanced, scenarios.