[Paper Review] Modular Gaussian Processes for Transfer Learning

Pablo Moreno-Muñoz, Antonio Artés-Rodrı́guez+1 more|arXiv (Cornell University)|Oct 26, 2021

Gaussian Processes and Bayesian InferenceComputer Science37 references1 citations

TL;DR

This paper introduces a modular Gaussian process framework for transfer learning that enables building meta-GP models from pre-trained, data-free GP modules. By leveraging variational inference and KL divergence-based integral augmentation, the method avoids retraining on original data, reduces computational cost, and supports multi-output and heterogeneous likelihoods—achieving state-of-the-art performance on regression, classification, and multi-task learning benchmarks without revisiting training data.

ABSTRACT

We present a framework for transfer learning based on modular variational Gaussian processes (GP). We develop a module-based method that having a dictionary of well fitted GPs, one could build ensemble GP models without revisiting any data. Each model is characterised by its hyperparameters, pseudo-inputs and their corresponding posterior densities. Our method avoids undesired data centralisation, reduces rising computational costs and allows the transfer of learned uncertainty metrics after training. We exploit the augmentation of high-dimensional integral operators based on the Kullback-Leibler divergence between stochastic processes to introduce an efficient lower bound under all the sparse variational GPs, with different complexity and even likelihood distribution. The method is also valid for multi-output GPs, learning correlations a posteriori between independent modules. Extensive results illustrate the usability of our framework in large-scale and multi-task experiments, also compared with the exact inference methods in the literature.

Motivation & Objective

• To address the limitations of centralized data reprocessing in transfer learning, especially in large-scale and privacy-sensitive settings.
• To eliminate the need to revisit original training data when combining models for new tasks.
• To enable efficient, scalable, and privacy-preserving transfer learning using only pre-trained GP module parameters.
• To support multi-output and heterogeneous likelihood models (e.g., Gaussian, Bernoulli) without retraining.
• To preserve learned uncertainty metrics across tasks through module-based composition.

Proposed method

• The framework uses sparse variational Gaussian processes as modular components, each storing only hyperparameters, pseudo-inputs, and posterior densities.
• A novel lower bound is derived using augmented integral operators based on the Kullback-Leibler divergence between stochastic processes.
• Meta-GP models are constructed by combining independent GP modules without accessing original data, using only their variational parameters.
• The method supports heterogeneous likelihoods by applying deterministic mappings (e.g., sigmoid) to link GP outputs to likelihood parameters.
• For multi-output learning, correlations between modules are learned a posteriori, enabling joint inference across different output types.
• The approach is scalable and computationally efficient, with complexity independent of original dataset size after module training.

Experimental results

Research questions

• RQ1 Can GP models be reused for new tasks without retraining on original data, preserving learned uncertainty and performance?
• RQ2 How can a principled, scalable, and privacy-preserving transfer learning framework be built using only pre-trained GP modules?
• RQ3 To what extent can modular GP models generalize across diverse tasks with heterogeneous likelihoods (e.g., regression and classification)?
• RQ4 How does the performance of meta-GP models compare to exact inference and state-of-the-art methods in large-scale and multi-task settings?
• RQ5 Can the framework effectively learn correlations between independent modules in multi-output scenarios?

Key findings

• The modular GP framework achieved a test NLPD of 7.21±0.04 on the banana dataset, outperforming the baseline variational GP (7.29±7.85×10−4) due to increased effective inducing points.
• On the US airline delays dataset, the meta-MOGP reduced prediction error by only ~1% compared to individual GP modules and the variational MOGP baseline, demonstrating robustness.
• For the London household dataset, the meta-MOGP achieved NLPD=4.18±0.06 in regression and NLPD=4.78±0.03 in classification, with improved performance on binary outputs compared to standalone modules.
• The method successfully enabled data-free meta-learning on MNIST pixel-wise classification, achieving accurate predictions without revisiting training data.
• In multi-output experiments, the framework maintained strong predictive performance across heterogeneous outputs (Gaussian and Bernoulli), with minimal degradation in error metrics.
• The framework demonstrated scalability and efficiency, with computational cost independent of original dataset size after module training, making it suitable for large-scale and continual learning settings.