[Paper Review] A Framework for Interdomain and Multioutput Gaussian Processes

Mark van der Wilk, Vincent Dutordoir+4 more|arXiv (Cornell University)|Mar 2, 2020

Gaussian Processes and Bayesian Inference42 references63 citations

TL;DR

The paper introduces a modular, scalable variational framework that unifies interdomain and multioutput Gaussian processes and provides a GPflow-based implementation for flexible, testable GP inference and deep GP architectures.

ABSTRACT

One obstacle to the use of Gaussian processes (GPs) in large-scale problems, and as a component in deep learning system, is the need for bespoke derivations and implementations for small variations in the model or inference. In order to improve the utility of GPs we need a modular system that allows rapid implementation and testing, as seen in the neural network community. We present a mathematical and software framework for scalable approximate inference in GPs, which combines interdomain approximations and multiple outputs. Our framework, implemented in GPflow, provides a unified interface for many existing multioutput models, as well as more recent convolutional structures. This simplifies the creation of deep models with GPs, and we hope that this work will encourage more interest in this approach.

Motivation & Objective

• Motivate the need for modular, scalable GP inference suitable for large-scale problems and deep GP integration.
• Present a unified mathematical framework combining interdomain approximations with multioutput GP models.
• Show how the framework supports efficient variational inference and minibatch training for GPs.
• Provide a software implementation in GPflow that enables rapid experimentation with interdomain and multioutput structures.

Proposed method

• Describe Gaussian process notation and manipulation for single- and multioutput settings.
• Explain interdomain Gaussian process concepts including interdomain observations and inducing variables.
• Develop variational inference for multioutput GPs and derive the corresponding posterior and marginal likelihood forms.
• Present the unified software framework in GPflow with interfaces for interdomain approximations and multioutput models.
• Demonstrate methods such as Linear Model of Coregionalization and image convolutional GPs within the framework.

Experimental results

Research questions

• RQ1 How can interdomain approximations be integrated with multioutput Gaussian processes in a scalable variational framework?
• RQ2 Can the proposed framework unify and simplify existing multioutput GP models and enable new architectures (e.g., deep GPs with convolutional structure)?
• RQ3 What is the impact of interdomain inducing variables on computational complexity and inference efficiency in MOGPs?
• RQ4 How can the framework facilitate rapid experimentation and deployment of multioutput GP models in practical settings?

Key findings

• The framework enables scalable approximate inference for interdomain and multioutput GPs with complexity reductions suitable for large datasets.
• GPflow provides a unified interface that covers many existing multioutput models and supports convolutional structures.
• The approach supports variational inducing variables and minibatch training to reduce training cost.
• The framework generalizes several models, including the Linear Model of Coregionalization and image-based GP constructions.
• It aims to ease the development of deep models that incorporate GPs by removing bespoke derivations for each variation.