[Paper Review] Meta-Learning Mean Functions for Gaussian Processes

Vincent Fortuin, Heiko Strathmann+1 more|arXiv (Cornell University)|Jan 23, 2019

Gaussian Processes and Bayesian Inference40 references20 citations

TL;DR

This paper proposes meta-learning the mean function of Gaussian processes (GPs) to improve performance on low-data tasks, using deep neural networks to parameterize the mean function and optimizing it across related meta-tasks. Empirical results show that learning the mean function consistently improves predictive performance over standard GP baselines and kernel-only meta-learning, especially in real-world medical time series tasks.

ABSTRACT

When fitting Bayesian machine learning models on scarce data, the main challenge is to obtain suitable prior knowledge and encode it into the model. Recent advances in meta-learning offer powerful methods for extracting such prior knowledge from data acquired in related tasks. When it comes to meta-learning in Gaussian process models, approaches in this setting have mostly focused on learning the kernel function of the prior, but not on learning its mean function. In this work, we explore meta-learning the mean function of a Gaussian process prior. We present analytical and empirical evidence that mean function learning can be useful in the meta-learning setting, discuss the risk of overfitting, and draw connections to other meta-learning approaches, such as model agnostic meta-learning and functional PCA.

Motivation & Objective

• Address the underexplored challenge of learning the mean function in meta-learning for Gaussian processes, despite its potential to encode prior knowledge.
• Investigate whether meta-learning the mean function can improve generalization in low-data regimes, particularly when kernel learning alone is insufficient.
• Mitigate overfitting risks in mean function learning by leveraging meta-learning across multiple related tasks, rather than standard supervised optimization.
• Draw connections between the proposed method and established meta-learning frameworks like MAML and functional PCA to strengthen theoretical grounding.
• Demonstrate the practical utility of mean function meta-learning on real-world medical time series prediction, where data scarcity is a major challenge.

Proposed method

• Parameterize the GP mean function using a deep neural network, enabling flexible, data-driven prior encoding.
• Optimize the mean function parameters via meta-learning by minimizing the expected negative log marginal likelihood across a set of meta-tasks.
• Use a meta-learning framework where each meta-task is a small dataset from a related distribution, and the target task is a new, low-data task.
• Train the mean function using a bilevel optimization scheme: inner loop performs GP inference on each meta-task, outer loop updates the meta-parameters via gradient descent.
• Combine meta-learned mean functions with meta-learned kernel functions for joint prior learning, improving flexibility and performance.
• Apply the method to both synthetic benchmarks and real-world medical time series data from the PhysioNet 2012 Challenge, using standard evaluation metrics like RMSE and MAE.

Experimental results

Research questions

• RQ1 Can meta-learning the mean function of a Gaussian process lead to improved performance on low-data prediction tasks compared to standard GP or kernel-only meta-learning?
• RQ2 What are the risks of overfitting when learning the mean function in a meta-learning setting, and how do they compare to standard supervised learning?
• RQ3 In what conditions is meta-learning the mean function particularly beneficial, especially when compared to kernel function meta-learning?
• RQ4 How does the proposed method relate to existing meta-learning frameworks such as MAML and functional PCA?
• RQ5 Can mean function meta-learning provide consistent gains in real-world, high-impact applications such as medical time series prediction?

Key findings

• Mean function meta-learning consistently outperforms GP models with fixed or kernel-only learned priors on benchmark regression tasks, with significant improvements in RMSE and MAE.
• On the PhysioNet 2012 medical time series dataset, models with meta-learned mean functions achieved lower RMSE (e.g., 1.58 ± 0.10 for BUN) compared to kernel-only or fixed-mean baselines.
• The method shows reduced overfitting risk compared to standard supervised optimization of the mean function, as demonstrated by stable performance across meta-tasks.
• Empirical results show that learning both the mean and kernel functions yields the best performance, with the mean function contributing significantly to predictive accuracy.
• The approach demonstrates strong empirical gains on diverse tasks, including synthetic functions and real-world clinical data, validating its generalizability.
• Theoretical and empirical connections to MAML and functional PCA suggest that mean function meta-learning aligns with broader principles of inductive bias transfer in meta-learning.