[Paper Review] Federated Gaussian Process: Convergence, Automatic Personalization and Multi-fidelity Modeling

Xubo Yue, Raed Al Kontar|arXiv (Cornell University)|Nov 28, 2021

Gaussian Processes and Bayesian Inference76 references4 citations

TL;DR

This paper proposes Federated Gaussian Process Regression (FGPR), a novel framework that enables privacy-preserving, decentralized learning using Gaussian processes via federated averaging and stochastic gradient descent. FGPR achieves convergence to a critical point of the global log-likelihood and enables strong personalization by jointly learning a global GP prior across clients, outperforming state-of-the-art methods in multi-fidelity modeling with uncertainty quantification.

ABSTRACT

In this paper, we propose exttt{FGPR}: a Federated Gaussian process ($\mathcal{GP}$) regression framework that uses an averaging strategy for model aggregation and stochastic gradient descent for local client computations. Notably, the resulting global model excels in personalization as exttt{FGPR} jointly learns a global $\mathcal{GP}$ prior across all clients. The predictive posterior then is obtained by exploiting this prior and conditioning on local data which encodes personalized features from a specific client. Theoretically, we show that exttt{FGPR} converges to a critical point of the full log-likelihood function, subject to statistical error. Through extensive case studies we show that exttt{FGPR} excels in a wide range of applications and is a promising approach for privacy-preserving multi-fidelity data modeling.

Motivation & Objective

• To develop a federated learning framework for Gaussian processes that preserves data privacy while enabling collaborative model learning.
• To establish theoretical convergence of the federated GP regression framework to a critical point of the global log-likelihood function.
• To enable automatic personalization by learning a shared global GP prior that is conditioned on local client data.
• To demonstrate the effectiveness of FGPR in multi-fidelity data modeling with uncertainty quantification and privacy preservation.

Proposed method

• FGPR uses Federated Averaging (FedAvg) for global model aggregation, combining local model updates from clients.
• Local client computations are performed using stochastic gradient descent (SGD) on mini-batches of data.
• The global model learns a shared GP prior across all clients, which is then personalized via conditioning on local data.
• Theoretical analysis establishes convergence to a critical point of the full log-likelihood under conditions on kernel eigendecay (e.g., Matérn, RBF kernels).
• The framework handles correlated data points via a joint Gaussian process formulation, avoiding biased mini-batch gradients.
• Statistical error bounds are derived based on client mini-batch size and kernel properties.

Experimental results

Research questions

• RQ1 Can federated learning be successfully extended to Gaussian processes, which inherently model correlations across all data points?
• RQ2 Does the proposed FGPR framework converge to a critical point of the global log-likelihood function under realistic assumptions?
• RQ3 Can FGPR achieve effective personalization by learning a shared global GP prior and conditioning it on local data?
• RQ4 How does FGPR perform in multi-fidelity modeling compared to state-of-the-art methods?
• RQ5 What is the impact of mini-batch size and kernel structure on the statistical error and convergence of FGPR?

Key findings

• FGPR converges to a critical point of the full log-likelihood function, with convergence error bounded by statistical fluctuations dependent on mini-batch size.
• The framework achieves strong personalization by learning a global GP prior that is refined through local data conditioning.
• FGPR demonstrates superior performance in multi-fidelity modeling, outperforming existing methods in uncertainty quantification and data efficiency.
• Theoretical convergence holds for kernels with exponential or polynomial eigendecay, including Matérn and RBF kernels.
• Statistical error in convergence is bounded by terms proportional to $\frac{\log M}{M}$, where $M$ is the number of data points per client.
• The analysis provides the first theoretical extension of federated learning beyond empirical risk minimization to correlated stochastic processes like GP.