[Paper Review] GPflow: A Gaussian process library using TensorFlow

Alexander Matthews, Mark van der Wilk+6 more|arXiv (Cornell University)|Oct 27, 2016

Gaussian Processes and Bayesian InferenceComputer Science305 citations

TL;DR

GPflow is a TensorFlow-based Gaussian process library that uses variational inference as the main approximation method, supports GPU acceleration, and emphasizes testing and open-source development with a Python front end similar to GPy.

ABSTRACT

GPflow is a Gaussian process library that uses TensorFlow for its core computations and Python for its front end. The distinguishing features of GPflow are that it uses variational inference as the primary approximation method, provides concise code through the use of automatic differentiation, has been engineered with a particular emphasis on software testing and is able to exploit GPU hardware.

Motivation & Objective

• Motivate a fast, scalable, and verifiably correct Gaussian process library.
• Provide a concise, extensible Python interface inspired by GPy.
• Leverage automatic differentiation and GPU hardware to ease implementation and speed up computations.
• Contribute to TensorFlow by adding needed GP capabilities (e.g., Cholesky differentiation) and GPU solutions.
• Ensure open-source availability with robust testing and documentation.

Proposed method

• Use TensorFlow to implement core GP computations and automatic differentiation for gradients.
• Adopt variational inference as the primary approximation to handle non-conjugacy and scaling.
• Provide a clean object-oriented Python front end with a shared base class for inference methods.
• Contribute new TensorFlow operations (e.g., Cholesky differentiation) and GPU-accelerated linear algebra to enable scalable GP inference.
• Offer a suite of inference options including exact inference where possible and various variational/MCMC methods (e.g., HMC) for non-Gaussian likelihoods.
• Maintain high code coverage (approximately 99%) and continuous integration for quality assurance.

Experimental results

Research questions

• RQ1 How can Gaussian process inference be made scalable to large datasets using variational sparsity?
• RQ2 Can TensorFlow-based GPflow provide efficient and accurate variational, MCMC, and exact inference within a unified framework?
• RQ3 What software engineering practices (testing, open source, GPU support) improve usability and extensibility of GP libraries?
• RQ4 How does GPflow compare to existing libraries (e.g., GPy) in terms of performance and usability when deployed on CPU vs GPU?

Key findings

• GPflow supports exact inference where possible and various scalable variational methods (variational sparsity).
• It relies on automatic differentiation to minimize gradient coding, and runs on GPUs for speedups.
• GPflow’s Python front end is object-oriented and shares design lineage with GPy, but its core computations run in TensorFlow.
• The library demonstrates high test coverage, achieving about 99% in their reporting.
• Timed experiments show GPU acceleration yields considerable speedups over CPU-only runs for a MNIST-based variational GP classifier.
• GPflow code and front end are released as open-source under the Apache 2.0 license.