[Paper Review] Convolutional Gaussian Processes

Mark van der Wilk, Carl Edward Rasmussen+1 more|arXiv (Cornell University)|Sep 6, 2017

Gaussian Processes and Bayesian InferenceComputer Science14 references53 citations

TL;DR

This paper introduces convolutional kernels for Gaussian processes using an inter-domain inducing-point approximation, enabling scalable, convolution-aware GP inference for image data and combining convolutional and RBF components via marginal likelihood.

ABSTRACT

We present a practical way of introducing convolutional structure into Gaussian processes, making them more suited to high-dimensional inputs like images. The main contribution of our work is the construction of an inter-domain inducing point approximation that is well-tailored to the convolutional kernel. This allows us to gain the generalisation benefit of a convolutional kernel, together with fast but accurate posterior inference. We investigate several variations of the convolutional kernel, and apply it to MNIST and CIFAR-10, which have both been known to be challenging for Gaussian processes. We also show how the marginal likelihood can be used to find an optimal weighting between convolutional and RBF kernels to further improve performance. We hope that this illustration of the usefulness of a marginal likelihood will help automate discovering architectures in larger models.

Motivation & Objective

• Incorporate convolutional structure into Gaussian processes to better generalise on high-dimensional inputs like images.
• Develop an inter-domain inducing point approximation tailored to a convolutional kernel for scalable inference.
• Explore variations of the convolutional kernel and assess their performance on image datasets.
• Demonstrate how marginal likelihood can automatically weight convolutional versus RBF components to optimize performance.

Proposed method

• Construct a patch-based convolutional GP by applying a patch-response function g with a GP prior and summing patch responses to form f, yielding a convolutional kernel.
• Adopt a variational sparse GP framework with inducing inputs and inter-domain inducing variables to enable scalable inference (ELBO optimization).
• Derive inter-domain covariances for the convolutional kernel to place inducing points in patch space, reducing kernel evaluation cost.
• Explore translation-invariant, weighted, and multi-channel convolutional kernels, and combine with an RBF kernel using a joint GP with marginal likelihood.
• Extend kernels to color images with approaches including color-patch and multi-channel variants; apply to MNIST and CIFAR-10 datasets.

Experimental results

Research questions

• RQ1 Can a convolutional kernel be embedded in Gaussian processes via inter-domain inducing points to achieve scalable, patch-based inference for images?
• RQ2 How do different convolutional kernel variants (translation-invariant, weighted, color-aware) compare on standard image benchmarks like MNIST and CIFAR-10?
• RQ3 Does marginal likelihood effectively balance convolutional structure with non-convolutional components (e.g., RBF) to improve predictive performance?
• RQ4 What are the trade-offs in using inter-domain inducing variables versus standard inducing points for convolutional GP kernels?

Key findings

• Translation-invariant convolutional GP with patch-space inducing points yields competitive results on MNIST with fewer inducing points than an RBF baseline.
• Weighted convolutional kernels significantly improve MNIST error to 1.22% (NLPP 0.048) compared to 1.90% (NLPP 0.068) for RBF.
• A combination of weighted convolutional and RBF components further reduces MNIST error to 1.17% (NLPP 0.039).
• For full MNIST, the weighted convolutional kernel outperforms translation-invariant and RBF kernels, with the mixture achieving the best NLPP and lower error.
• On CIFAR-10, multi-channel convolutional kernels substantially improve test error over RBF, achieving 35.4% vs 48.6% (RBF).
• CIFAR-10 results indicate convolutional structure provides significant gains, particularly when color interactions are modelled.