[Paper Review] GPflux: A Library for Deep Gaussian Processes

Vincent Dutordoir, Hugh Salimbeni+8 more|arXiv (Cornell University)|Apr 12, 2021

Gaussian Processes and Bayesian Inference16 references7 citations

TL;DR

GPflux is a Python library that enables Bayesian deep learning with deep Gaussian processes (DGPs) by integrating GPflow and Keras, offering modular, extensible, and efficient implementations of state-of-the-art DGP models and inference schemes. It supports hybrid architectures combining Bayesian and standard neural network layers within a unified, production-ready framework with full Keras compatibility.

ABSTRACT

We introduce GPflux, a Python library for Bayesian deep learning with a strong emphasis on deep Gaussian processes (DGPs). Implementing DGPs is a challenging endeavour due to the various mathematical subtleties that arise when dealing with multivariate Gaussian distributions and the complex bookkeeping of indices. To date, there are no actively maintained, open-sourced and extendable libraries available that support research activities in this area. GPflux aims to fill this gap by providing a library with state-of-the-art DGP algorithms, as well as building blocks for implementing novel Bayesian and GP-based hierarchical models and inference schemes. GPflux is compatible with and built on top of the Keras deep learning eco-system. This enables practitioners to leverage tools from the deep learning community for building and training customised Bayesian models, and create hierarchical models that consist of Bayesian and standard neural network layers in a single coherent framework. GPflux relies on GPflow for most of its GP objects and operations, which makes it an efficient, modular and extensible library, while having a lean codebase.

Motivation & Objective

• To address the lack of actively maintained, extensible, and open-source libraries for deep Gaussian processes (DGPs) in Bayesian deep learning.
• To provide reusable, modular components for researchers to implement novel DGP models and inference schemes efficiently.
• To enable practitioners to build, train, and deploy hybrid Bayesian and deep neural network models using standard deep learning tooling.
• To reduce the complexity of implementing DGPs by abstracting mathematical subtleties such as multivariate Gaussian conditioning and index bookkeeping.
• To ensure high code quality and usability through continuous integration, over 97% test coverage, and comprehensive documentation.

Proposed method

• GPflux is built on top of TensorFlow and integrates with Keras, enabling seamless use of deep learning ecosystem tools like model compilation, callbacks, and model saving.
• It leverages GPflow for core Gaussian process operations, ensuring efficiency and modularity while maintaining a lean codebase.
• The library provides specialized layers such as GPLayer for DGP components, LatentVariableLayer for augmented input spaces, and ConvolutionalGP layers for structured data.
• It uses TensorFlow Probability’s DistributionLambda to handle distributional outputs from GP layers, enabling reparameterization for end-to-end training.
• The framework supports hybrid models combining GPflux layers with standard Keras layers (e.g., Dense, Conv2D), allowing flexible, hierarchical model design.
• Inference is based on variational approximation via sparse inducing points, with the ELBO loss decomposed into data-fit and KL complexity terms handled automatically by the layers.

Experimental results

Research questions

• RQ1 How can deep Gaussian processes be implemented in a modular, extensible, and production-ready way within a modern deep learning ecosystem?
• RQ2 What architectural abstractions are necessary to simplify the implementation of novel DGP models and inference schemes?
• RQ3 How can Bayesian deep learning models combining GPs and standard neural networks be trained efficiently using existing deep learning tooling?
• RQ4 To what extent can existing deep learning frameworks like Keras be extended to support full Bayesian inference with Gaussian processes?
• RQ5 How can mathematical complexities in multilayered, multi-output GPs—such as index tracking and infinite basis functions—be abstracted for practical use?

Key findings

• GPflux provides a fully open-source, actively maintained library for deep Gaussian processes, filling a critical gap in the Bayesian deep learning ecosystem.
• The library achieves over 97% test coverage and is compatible with Keras, enabling integration with established deep learning workflows such as TensorBoard monitoring and model checkpointing.
• Hybrid models combining GPflux layers with standard Keras layers can be constructed using Sequential or custom models, supporting end-to-end training with standard optimizers.
• The framework successfully abstracts complex mathematical operations such as multivariate Gaussian conditioning and reparameterization, reducing implementation burden.
• GPflux enables researchers to build and evaluate novel DGP models and inference schemes with minimal boilerplate, thanks to its modular and composable design.
• The library supports advanced DGP components like latent variable layers and convolutional GPs, facilitating modeling of complex, non-Gaussian data distributions.